Disabling any visual effects from the "Appearance" section seems to disable all compositing. When there's none of that going on the video is almost smooth, there's still a few "hitches". What's interesting is that the mplayer status line in the console only shows a couple of dropped frames when the video is started, it doesn't increment when these hitches happen. Reported CPU usage for the video decoder over the entire video is between 40 and 45%.
I installed the compiz settings application and ran the benchmark plugin. At the composited desktop with nothing running it barely managed 30fps. With the 720p video running it drops down to 20-22fps. Dragging the video window, and having it wobble, drops the FPS into the high teens.
Using either the gl or gl2 mplayer output plugins increases the CPU usage greatly. mplayer reports that the "video out" cpu usage is around 35-40% and the video is very jumpy. The dropped frame count only increments occasionally, despite the jumpy video.
I re-ran the same tests on the GeForce 7600GS. I know it's a far more powerful GPU, but I figured it couldn't hurt.
FPS at the bare desktop was reported as being between 45 and 50 and nothing would cause a drop. I played with multiple compiz effects, even ran additional OpenGL applications, but it wouldn't budge. The video stayed smooth throughout.
I wondered if the Intel problems might have been related to the fact that the onboard GPU was only getting 32MB of the system memory allocated to it. The xorg log showed it was gaining an additional 256MB, but I went into the BIOS to try and change it anyway.
There were 2 settings related to video RAM.
This could be set to "Lite" or "Paranoid". When set to "Lite", no matter how much memory was allocated, the X server and Intel driver would segfault on startup, immediately after trying to initialize UXA.
When set to Paranoid, the X server would hard-lock the PC. It became unresponsive to keyboard input and stopped replying to network pings. Hitting restart was the only option.
This setting directly affected the RAM allocated to the agpgart, nothing else.
I don't know what else to test and I can't explain why the video performance is so different between the GPU's. Yes the 7600GS is more powerful, but surely as the CPU can decode the 720p video with no problems it can't require that much GPU power to display it on the screen. It's not like I'm resizing it or anything, it's playing 1:1 in a window.
If it weren't for the huge price difference, and general difficulty in finding a laptop with a 9300M GS / 9400M in, I'd have given up already =(
Thanks again all