Now modern X might be completely different and not even allow for the distributed load concept any more. And perhaps the VNC-like solution is the best going forward (or maybe something better can be created? something GPUDirect-like?), but I can understand the firestorm early on when the whole concept of networking was sort of dismissed as a sideshow that could be tacked on later if somebody ever got around to being interested. I think some of that controversy has died down as communication has gotten better.
But certainly for our UNIX and Linux systems X network capability was a critical component of our environment.