I found the issue. It's a Kernel Mode setting permissions issue with /dev/dri/card*
Most of my testing was done on VirtualBox, so the Framebuffer part got a better testing then.
Anyway it seems that with the fact that the permissions on the file is supposed to be granted by console-kit or the login manager, so I made a commit to add the plugdev users group permissions to access that file similar to the way I do it to /dev/fb*.
I don't know if there's a more secure way to do this, as any local user that can log in will be able to see /dev/dri/card0
I'm really sorry about this. I guess I should push a new ISO with the fix.