Linux Sees A New Attempt At Threaded Console Printing

[oiaohm]

Its a mixed bag. Threads could be more robust they could also introduce a lot of bug. And performance too. Too many threads can cause latency issues and other bugs.

Single threaded system/global lock for console has serous problems with performance and robustness already. Something to be aware is the new thread console is using what has been used in the logging system of the Linux kernel over decade.

https://lore.kernel.org/lkml/2022091...linutronix.de/ is a good read spotting the key warning about the existing simple is simple to miss.

1) The blurry semantics of console lock which triggers unpleasant memories of BKL. That in turn inspired me to flag the new consoles
with CON_NOBKL and use the nobkl theme throughout the series as I could not come up with a better name.

2) The assumption that seperating a printk thread out from console lock, but at the same time partially depending on console lock. That's not
really working out.

3) The operation of consoles and printk threads was depending solely on global state and that state is on/off so it does not really qualify as
stateful and is therefore not really useful to create a stable mechanism.

Do note here that existing system is half threaded as mentioned in point 2.

In fact there is not a single regular (non-early) console driver today which is reentrancy safe under all circumstances, which enforces that NMI context is excluded from printing directly. TBH, that's a sad state of affairs.

When you have hardware console devices bricking themselves without notice and need you to run reset functions so they keep on working you already have as problem with the existing system. There is also the cases where a CPU core for some reason magically locks up if that happened to be your console at the moment there will be no logging about it.

The reality we already have a lot of console bugs. It very possible that going to a full thread console without locks will remove a lot of bugs. Latency issue exists already with the existing console. The RT kernel mode of Linux issue with the existing is the amount of latency you spend getting console messages from one CPU core to another due to locking.

Console Threads done with better IPC can be better performance than the existing Console code. Remove the locks you remove dead locking.. Console dead locking is a really bad thing.

The new attempt at threaded Console kind of deceptive. The reality is the Linux kernel has thread console already based heavily on locking. As in you need to take out console lock before you can do anything.

All console state operations rely solely on atomic*_try_cmpxchg() so they work in any context.

This new threaded console moves away from locking.

Just ue a atomic list as a command buffer or something.

Yes to exactly what you wrote is in the new attempted threaded console is.

Having a threaded console is not new to Linux. The current console design in the linux kernel at core goes back to 0.1 Linux as in 1991. Atomic operations did not exist in CPUs of that time. Yes the i386. Linux kernel removed the BKL but has kept the global console lock.

Mixture of single threaded/multi threaded that using locks cause is normally a very bad thing. Particularly when we have had atomic since the i486.

The current Linux console is broken. The new threaded console design could fix lots of broken.

Lot of ways for normal people to understand is not to call this change "threaded console printing" but remove of the "Console Big Kernel Lock" so change over to using atomic between console threads correctly to bring robustness and performance.

Yes just like the migrating away from the "Big Kernel Lock" there is going to be a few issues here or there but long term removal of the "Console Big Kernel Lock" will be gain.

There are likely other per subsystem "Big Kernel Lock" replacements that need to be exterminated like the "Console Big Kernel Lock" still in the Linux kernel.

Yes you could also class this patch set as attempting to complete "Big Kernel Lock removal" as well.

[cj.wijtmans]

The circle buffers/ring buffers in threaded console printing are atomic lists. I agree that extra features in kernel not great. But the Linux kernel console has not been as robust as it could have been. Some of the extra complexity being added will detect cases when a console driver that should be output console information has locked up and is no longer output information. Threading the console system allows adding particular forms of error detection you cannot add otherwise.

Threaded console is mixture of performance and improved robustness. Yes means to change console output driver while system is running is something that threaded console code is going to bring as well. Historically under Linux if the console output driver dies only way to fix it is restart system completely losing the logging data in the process.

Its a mixed bag. Threads could be more robust they could also introduce a lot of bug. And performance too. Too many threads can cause latency issues and other bugs.

[oiaohm]

Great more thread pollution just what we needed to introduce more bugs and performance issues. This approach is very wrong in trying to make printk more atomic. Just ue a atomic list as a command buffer or something.

This is wrong. A kernels console should be small simple robust bug free and ANSI/posix. If you require a more complex terminal use a modern desktop. If not present locally then use SSH or whatever software/hardware connection.

The circle buffers/ring buffers in threaded console printing are atomic lists. I agree that extra features in kernel not great. But the Linux kernel console has not been as robust as it could have been. Some of the extra complexity being added will detect cases when a console driver that should be output console information has locked up and is no longer output information. Threading the console system allows adding particular forms of error detection you cannot add otherwise.

Threaded console is mixture of performance and improved robustness. Yes means to change console output driver while system is running is something that threaded console code is going to bring as well. Historically under Linux if the console output driver dies only way to fix it is restart system completely losing the logging data in the process.

[cj.wijtmans]

Great more thread pollution just what we needed to introduce more bugs and performance issues. This approach is very wrong in trying to make printk more atomic. Just ue a atomic list as a command buffer or something.

There needs to be a totally new console. Something modern, Unicode aware, hardware accel, etc.

This is wrong. A kernels console should be small simple robust bug free and ANSI/posix. If you require a more complex terminal use a modern desktop. If not present locally then use SSH or whatever software/hardware connection.

[oiaohm]

It won't only be useful for scrollback. An additional buffer at that lower level will also cut down the locking and IPC for logging and dumps which is very relevant to embedded usage. As for the additional memory overhead, even if it wasn't just a few bytes, you could still make the whole thing toggle-able as a build or run flag. Another possible advantage is doing something like frame dropping to console output by delaying printout when constraint until the buffer fills up. That's something that would definitely have uses in real time embedded.

That is no. The current linux kernel circle buffers does everything you raise as issues.

https://www.systutorials.com/docs/linux/man/2-syslog/

Scrollback is maintaining information after it been already output majority of you embedded cases not need this.

Circle buffer already drops the IPC about as much as you can. Circle buffer already handles the case that printout is constrained by overwriting the oldest entries.

Do note that logging and dumps do have their own circle buffer on top of the console circle buffers.

Yes circle buffer is for the likes of logging would be very much what you would be thinking about as a scrollback buffer. Note the logging circle buffer is not the same as the user console scroll back buffer and so on.

Yes scrollback buffer being it own thing console driver makes sense. Shared code for implement it could be possible. Why would logging and like the serial console be different buffers the the differences in delay. What is acceptable for a local running logging program could be very different to sending to screen or sending over serial port.

Single scrollback buffer does not make much sense. You were not thinking that logging has it own circle buffer already. Console drivers implement the circle buffers they need.

The core of this threaded console printing is feeding stuff out into the circle buffers/ringbuffers of the outputs. Some of these includes possible code to restart a output console if the ring buffer assigned to the output console gets full because if the this happened the output console has most likely failed.

See the problem you said scrollback buffer. This is a buffer keeping X number of entries normally a max number of entries at all times. There is a need for a output console to have a buffer that min possible entries. As in only the entries the console has not printed out yet on the buffer.

Scrollback can be a ring buffer that loops the end of the buffer around. Problem with scrollback buffer how do you detect issue with console output.

Thread console is using min filled ring buffer that max filled ring buffer is sign that the console is not keeping up so could be in trouble. Take a network/serial cable where someone pulled the cord and the messages stopped getting though for some reason.

Scrollback sounds like a nice feature with consoles. Back pressure detection is more important than scrollback in lots of cases. You don't do back pressure detection with scrollback buffer. Back pressure detection ring buffer if the buffer max value number of entries get reached this will trigger something. Yes back pressure buffer handles restricted amount of output delay.

Scrollback buffers being in like the logging as it own ring buffer that can loop around or in console drivers as items that can loop around this is fine. Of course this being feed by back pressure detection ring buffers. So that if the logging system is not transfering stuff to the logging buffers or the console driver is not transfering stuff for output problem is detectable.

Scrollback buffer is just one things you can create with a ring/circle buffer but its not the only thing. Back pressure detection ring buffers make more sense than Scrollback buffer for lots of use cases.

1) What is the point of sending messages to a console that are never getting printed?
2) How are you going to detect that messages are not getting printed?

Yes back pressure ring buffer will not detect all cases of console driver being dead but it will detect lot more than scrollback buffer.

[c117152]

There are a lot of embedded usages of Linux. Yes that embedded usage is not using screens. One of the biggest issues with the threaded console support that has delayed it implementation is a failure to get to serial port all the errors. When scrolling back you are stopping output.

There is a reason why scrollback buffer is at the console driver level. Usage where the scrollback buffer is not usable there is no point to it. You have data collection device on a serial port recording all the Linux kernel console output the data collection device provides the scroll back functionality not the Linux kernel.

Could generic scrollback buffer be added for those implementing console drivers to use yes it could but this would be out side the core console line.

Do note graphics capture you said additional copy into dedicate scrollback buffer. This is independent buffer to the back-buffer right that going to memory and resources right.

It won't only be useful for scrollback. An additional buffer at that lower level will also cut down the locking and IPC for logging and dumps which is very relevant to embedded usage. As for the additional memory overhead, even if it wasn't just a few bytes, you could still make the whole thing toggle-able as a build or run flag. Another possible advantage is doing something like frame dropping to console output by delaying printout when constraint until the buffer fills up. That's something that would definitely have uses in real time embedded.

[oiaohm]

But that's only how things used to be. Now that they're already changing the lower level buffers they can just throw in a backbuffer and a lock-less copy into an additional dedicated scrollback buffer like how lock-less screen capture for streaming works. It's common practice in graphics.

There are a lot of embedded usages of Linux. Yes that embedded usage is not using screens. One of the biggest issues with the threaded console support that has delayed it implementation is a failure to get to serial port all the errors. When scrolling back you are stopping output.

There is a reason why scrollback buffer is at the console driver level. Usage where the scrollback buffer is not usable there is no point to it. You have data collection device on a serial port recording all the Linux kernel console output the data collection device provides the scroll back functionality not the Linux kernel.

Could generic scrollback buffer be added for those implementing console drivers to use yes it could but this would be out side the core console line.

Do note graphics capture you said additional copy into dedicate scrollback buffer. This is independent buffer to the back-buffer right that going to memory and resources right.

[c117152]

https://lwn.net/Articles/881827/

The scroll back buffer is outside the printk to console stuff. console.h straight buffering code for output has never had anything todo with the scroll backbuffer. Console buffering for printk and the like is more of a ring buffer.

Scroll back buffer is functionality of the console driver itself if it happens to support it.

c117152 think about all the extra locking you would cause with a global scroll back buffer. Yes if your only console output off a device is going out a serial port that goes to storage there is no need for the kernel to have scrollback buffer as well.

Console.h buffer code is how to get the data to all the consoles. Not the place for user-friendly stuff like scroll back buffers.

But that's only how things used to be. Now that they're already changing the lower level buffers they can just throw in a backbuffer and a lock-less copy into an additional dedicated scrollback buffer like how lock-less screen capture for streaming works. It's common practice in graphics.

[andrei_me]

From the mailing list replies, this new patches seems to contains a well designed approach to the threaded printing, Linus haven't said anything bad about the patches, so this is good

[oiaohm]

There needs to be a totally new console. Something modern, Unicode aware, hardware accel, etc.

Like kmscon?

There is consoled in systemd source package that most distributions don't build or provide that was to be newer version of kmscon that end up not happening..