Any of Nokia's Maemo based Internet Tablets or the N900 and N9 use Xorg as it's display server and were plenty fast. The N9 being extremely slick. The first few builds of SailfishOS for the Jolla phones used it as well, but is now using Wayland. So yes, even old Xorg is fast enough for mobile devices. There never was any 'performance' issues on them. Especially after doing some nice transition tweaks, they were really smooth.
Unless Wayland and Mir become much faster than compositing X on small devices, tablets and phones will have continue to have real performance issues. That's why the first ubuntu phones used Android's "surfaceflinger" display server. I suppose a nonaccelerated X could be used with a QT-based Unity 2d fork, but all those tablets and phones were intended from the start for accelerated graphics.