As usual, you'll get HWE backports from 20.10+ included in the point-release images by default, and only apt install away in the archive, so even if xserver...