It's tricky to use as a root FS, because you can only setup grub on it from the GRUB bootloader itself, not while running Linux with grub-install. (I keep a GRUB floppy image on a USB drive, that I can boot with syslinux's memdisk. I also have it on my PXE netboot server.)
Also, XFS needs to be mounted with logbsize=256k to do well all around (at least on magnetic disks). However, putting that in /etc/fstab for the root fs doesn't do anything, because for the root fs it only does a mount -o remount to try to apply the options. So if only logbsize=256k was the default on systems with plenty of RAM... (with the default logbsize=32k, IIRC, you bottleneck on journaling metadata ops any time you write a lot of small files, e.g. extracting a tar). Be sure you use a recent xfsprogs, or manually specify lazy_count=1.
Anyway, wow, ext4 does great on solid-state disks! I wonder what the bottlenecks are in the other filesystems... Maybe CPU usage on the Atom? I know XFS is not a small piece of code, and it's definitely tuned for massive SMP systems, but maybe not for systems with such a slow CPU, esp. relative to the I/O latency of a solid state disk.