libwayland (server, client) does not have backends either, it just turns the binary protocol into nice C functions in a generic way.
Compositors can have backends, and Weston does too. Each compositor can load backends in any form they want. The stupid scenario is possible with compositors, unfortunately. If the hardware drivers do not implement what we think as standard APIs (say Linux KMS, EGL, and GBM; or OpenWF, hell even the /dev/fb framebuffer API), it's really a choice between "these particular compositors are the only Wayland servers that run on this exotic piece of hardware" vs. "no Wayland server runs on this hardware". Btw. I think you could run Weston with the generic fbdev backend on RPi, but you would get bored waiting for the screen to update---I never even tried.
Clients, applications, and toolkits on wayland do not have backends (in the sense we are talking about here), but they rely on standard APIs like EGL and OpenMAX to take advantage of hardware acceleration. This means that someone (usually a platform vendor or such) provides libraries that implement these standard APIs, and then applications link to them at runtime. Whether Wayland-enabled implementations of these standard APIs exist on a particular platform is a whole another question, but the Wayland client side does not require platform specific changes in applications or even toolkits.
However, an excellent question is, do standard APIs exist for everything the hardware could accelerate. That is the key to avoid making applications platform-specific.
In summary, if hardware does not come with the kind of drivers we expect, well, too bad. We may be able to work around that like with RPi, or maybe not. Sometimes it is possible to write new drivers the way we want them, and sometimes it is unrealistic.