Improving Video Streaming With GTK+ 3D Surfaces

[Nobu]

Problems observed (if any):
None, as far as playback goes.

GPU vendor, type, core clock speed, onboard / shared memory, bus:
nVidia, GeForce 7100, 350MHz, 128MiB / none, PCIe v1.0

CPU vendor, type speed, system memory:
AMD Athlon64 x2 3600+, 2.0Ghz, 1.5GB

OS version:
Ubuntu 11.04

GUI environment:
xfce4 & built-in compositing

Any extra information you may think is relevant:
It crashes (Segmentation fault) whenever I try to resize the window.
The video I used was a small (320x240) ogg. Anyone have a link to a bigger video I can download?
Edit: Oh yeah, using nouveau driver. ;-)

[pixelpusher]

Here are my results:

Compositing active: flickering on the rotating cube window. It seems to alternate between a part of my desktop and the cube animation. Although with a second video it alternates between solid black (maybe a part of an open terminal since I use black background?) and the cube animation. Also the alternating background image is always visible flickering even with other windows on top of GtkGLApp.

Without compositing everything works fine apart from the problems mentioned in the beggining.

Extra info:
catalyst drivers 10.11

This seems a common problem with the ATI drivers - with compositing disabled there is no flicker but speed is slow; enabled it seems the underlying graphics system forgets where the textures have been allocated and renders b0rked regions. It doesn't happen with nVidia drivers so (my guess) ATI's drivers aren't tracking XGL contexts properly.

Regards,
Jose.

[pixelpusher]

Source code released.

Hey folks, the source code has been released:

Source

Regards,
Jose.

[pixelpusher]

Hi folks, source is now available:

Source code

[nevion]

Gtk GL integration and behavior has always been weird. It's one of the reasons I went with Qt (on top of arguably better design / less boiler plate). It's official and integrated code.

I remember the hodgepodge of bindings a few years ago (I think there were at least 2) used to have some problems that some inside Gtk pals said prevented it's inclusion in Gtk itself and the whole situation was very unsavory. Upstream unofficially chose to ignore people who needed openGL or something to that effect (not that this was of malicious intent). This meant you really couldn't count on the GL libraries being well adopted (packages were not available for distro x) and they were buggy. GL itself is tricky if compositing window managers haven't taught you that, it's alot of work to get expected behavior out of all the drivers out there. But not being a well adopted library and buggy wrt end results (considering GL drivers being partially responsible) made using Gtk and openGL a real pain in the but circa 2006, a time I had real work I needed done and I regretted going with Gtk. I doubt I'll willingly go back to Gtk merely because of the shear amount of boiler plate that C++ takes care of for you automatically (sorry folks, I like being productive), but my GL related memories are icing on the cake.

Today I do more complex things and I do them easier, and I thank Qt for enabling. Sorry for being flame bait but I just don't understand why people are still driving Gtk forward anymore. In my eyes it's a waste of the developer's time and most people I know professionally abandoned it or never considered it because of Qt4's existence. What fuels your passion knowing that it's already been done well and is free in all senses?

[lienmeat]

pretty cool

On my laptop which has an intel 965gm in it, it segfaults on resize, but otherwise it runs well with a 720p video.

I'm running ubuntu 10.10 64 bit. I should note I had to install libgtkglext1 for it to run...you probably will to if you are running ubuntu 10.10...

I'll be posting later when I get home and test it on my ati 4850 with catalyst drivers.

[HokTar]

Works flawlessly here with a Mobility Radeon HD4650 (rv730) and r600g.

[devius]

This seems a common problem with the ATI drivers - with compositing disabled there is no flicker but speed is slow;

I should have been more clear. It's always slow. With composite on or off, with high bitrate or low bitrate, it's always the same 2fps of video playback in both windows.

[popper]

Hey folks, the source code has been released:

Source

Regards,
Jose.

Hi Jose, im curious and something i don't get, why are you Setting RGB when it's well known that all FFmpeg/'ffmpegcolorspace' indeed all video today are usually using YV12 as their output ?

"* Because of varying decoder formats (and our native OpenGL textures are RGB24) we add the 'ffmpegcolorspace' converter to the
* pipeline - this is not ideal when there is a GPU present that can perform the conversion much faster with fragement shaders.
*/
theGStVideoConverter = gst_element_factory_make ("ffmpegcolorspace", NULL);
gst_bin_add_many (GST_BIN (theGStVideoBin), theGStVideoConverter, theGStVideoSink, NULL);
/* Tell the converter element that we want raw rgb video at the end point. */
theGStCapsFilter = gst_caps_new_simple ("video/x-raw-rgb", NULL);
gst_element_link_filtered (theGStVideoConverter, theGStVideoSink, theGStCapsFilter);
gst_caps_unref (theGStCapsFilter);
/* Add a 'ghost' pad that will act to link as a dynamic pad when the video format is determined. */
theGStVideoPad = gst_element_get_static_pad (theGStVideoConverter, "sink");
gst_element_add_pad (theGStVideoBin, gst_ghost_pad_new ("sink", theGStVideoPad));
gst_object_unref (theGStVideoPad);
gst_bin_add(GST_BIN (theGStPipeline), theGStVideoBin);
"

as i understand it , if you set a 'video/x-raw-yuv,format=(fourcc)YV12 as the output then This OpenGL extension supports texture images stored in these YCbCr format's directly

http://www.opengl.org/registry/specs/MESA/ycbcr_texture.txt

"

and have you profiled the example code to acually see where this "2fps of video playback" bottleneck is happening BTW ?

[pixelpusher]

Hi Jose, im curious and something i don't get, why are you Setting RGB when it's well known that all FFmpeg/'ffmpegcolorspace' indeed all video today are usually using YV12 as their output ?

"* Because of varying decoder formats (and our native OpenGL textures are RGB24) we add the 'ffmpegcolorspace' converter to the
* pipeline - this is not ideal when there is a GPU present that can perform the conversion much faster with fragement shaders.
*/
theGStVideoConverter = gst_element_factory_make ("ffmpegcolorspace", NULL);
gst_bin_add_many (GST_BIN (theGStVideoBin), theGStVideoConverter, theGStVideoSink, NULL);
/* Tell the converter element that we want raw rgb video at the end point. */
theGStCapsFilter = gst_caps_new_simple ("video/x-raw-rgb", NULL);
gst_element_link_filtered (theGStVideoConverter, theGStVideoSink, theGStCapsFilter);
gst_caps_unref (theGStCapsFilter);
/* Add a 'ghost' pad that will act to link as a dynamic pad when the video format is determined. */
theGStVideoPad = gst_element_get_static_pad (theGStVideoConverter, "sink");
gst_element_add_pad (theGStVideoBin, gst_ghost_pad_new ("sink", theGStVideoPad));
gst_object_unref (theGStVideoPad);
gst_bin_add(GST_BIN (theGStPipeline), theGStVideoBin);
"

as i understand it , if you set a 'video/x-raw-yuv,format=(fourcc)YV12 as the output then This OpenGL extension supports texture images stored in these YCbCr format's directly

http://www.opengl.org/registry/specs/MESA/ycbcr_texture.txt

"

and have you profiled the example code to acually see where this "2fps of video playback" bottleneck is happening BTW ?

Hi,
The colour format you quote is not natively supported by the large majority of OpenGL cards as an internal hardware-based format. It's also a MESA - not OpenGL generic - extension. See this link for reliable internal texture formats.

As for the 2fps issue, my best guess (without OProfile information to hand on your setup) would be either a bottleneck in KDE's thread handoff or just plain slow texture transfer to the card. What bus is your HD4200 running on?

Regards,
Jose.