The Linux Kernel Has Been Forcing Different Behavior For Processes Starting With "X"

An ugly hack within the Linux kernel that has been in mainline for over three years has been called out. Due to a buggy X.Org Server / xf86-video-modesetting DDX, the Linux kernel has been imposing different behavior on whether a process starts with "X" and in turn disable the atomic mode-setting support.

Linux security researcher Jason Donenfeld (who is also well known for coming up with WireGuard), stumbled upon this ugly code hack within the kernel.

Donenfeld commented this weekend on the kernel mailing list:

This reverts 26b1d3b527e7 ("drm/atomic: Take the atomic toys away from X"), a rootkit-like kludge that has no business being inside of a general purpose kernel. It's the type of debugging hack I'll use momentarily but never commit, or a sort of babbies-first-process-hider malware trick.

The backstory is that some userspace code -- xorg-server -- has a modesetting DDX that isn't really coded right. With nobody wanting to maintain X11 anymore, rather than fixing the buggy code, the kernel was adjusted to avoid having to touch X11. A bummer, but fair enough: if the kernel doesn't want to support some userspace API any more, the right thing to do is to arrange for a graceful fallback where userspace thinks it's not available in a manageable way.

However, the *way* it goes about doing that is just to check `current->comm[0] == 'X'`, and disable it for only that case. So that means it's *not* simply a matter of the kernel not wanting to support a particular userspace API anymore, but rather it's the kernel not wanting to support xorg-server, in theory, but actually, it turns out, that's all processes that begin with 'X'.

Playing games with current->comm like this is obviously wrong, and it's pretty shocking that this ever got committed.

The commit to this kernel with the "X" first character check was made back in September 2019. The argument in that Linux kernel commit at the time was:

The -modesetting ddx has a totally broken idea of how atomic works:
- doesn't disable old connectors, assuming they get auto-disable like with the legacy setcrtc
- assumes ASYNC_FLIP is wired through for the atomic ioctl
- not a single call to TEST_ONLY

[In other words] the implementation is a 1:1 translation of legacy ioctls to atomic, which is a) broken b) pointless.

We already have bugs in both i915 and amdgpu-DC where this prevents us from enabling neat features.

If anyone ever cares about atomic in X we can easily add a new atomic level (req->value == 2) for X to get back the shiny toys.

Since these broken versions of -modesetting have been shipping, there's really no other way to get out of this bind.

The "good" news is that since then on user-space side back in 2019 the xf86-video-modesetting code went ahead and disabled atomic support by default. So technically if running an updated X.Org stack within the past three years, this kernel hack isn't necessary anymore since user-space is just then avoiding the atomic API.

We'll see if Linus Torvalds is okay with this hack being removed since after all it does go against his principle of "not breaking user-space" by then regressing the system if sticking to an outdated X.Org Server stack and wanting to run with a future kernel version. We'll see but surprising it took three years for this dirty code to be critiqued.