Linux Kernel Getting io_uring To Deliver Fast & Efficient I/O

[rene]

Problem with your idea its already been tried in the Linux kernel did not provide the performance boost anywhere near expected. Linux kernel is no where near as simple as you think it is.

https://lwn.net/Articles/755919/

bpfilter is one of the next generation of Linux kernel drivers. This is a mix of user space and kernel mode and bpf in kernel mode all in a single Linux .ko driver. Linux kernel is tuning into a strange form of hybrid.

Audit-able intentionally turing incomplete(as in fails the turing test) language that ebpf is that jit to native code by kernel to run in kernel space provides many times the performance boost bundling syscalls can this is shown by ebpf and bundle syscalls being use to attempt performance boost fuse under Linux. It due to the fact some basic logic can be performed kernel side complete event runs can be completed without any context switches with bpf.

Also you miss one of the big causes of context switching in microkernels. There are many operations that a driver performs that in fact need ring 0. This is not want this is need. IOPL (I/O Privilege level) flag does not provide to rings other than ring 0 the right to mess with memory permissions and other things drivers need to do with DMA driven hardware so this is happening a lot.

Microkernel core as ring 0 and drivers at like ring1/ring2 with IOPL flag you get performance wrecked by memory permission operations that must happen at ring 0 resulting in a mandatory context switch. so killing performance. This makes spectre performance loses look minor.

Microkernel core could run as hypervisor ring -1. Then each driver need to run a individual ring 0 vm then hypervisor transfer over head kills you. This results in Microkernel being a watchdog at ring -1 over a big blob monolith at ring 0 as this does perform.

CPU we have today are not designed to run microkernels effectively. Linux kernel hybred experiments might show way out.

https://access.redhat.com/articles/3311301

Big thing you have missed Linux kernel did add spectre and other cpu bug mitigation but then also includes flags to turn those off for those who want speed.

This is the hard bit design a OS you have security and performance that are both at time mutually exclusive you have to allow the user to choose what one they need more of. So the spectre overhead as a arguement for micro-kernel does not fly. Linus is willing to take a performance over head for security as long as it has a off flag to return the lost performance while decreasing the security. See the micro-kernel fail here its designed for security so you are stuck with the overhead if it suites your current problem or not..

a short note to a long story: i386 and later have an i/o permission bitmap for supposedly fine grained i/o permission control, also modern hardware is anyway not using classic i/o ports, with (nearly) everything memory mapped ring 3 drivers can drive the hardware just fine without any extra i/o context switching. Also QNX is quite fast, even was a decade ago, so it is not like it is impossible to do. Also more elegant architecture and algorithms can vastly improve performance, e.g. look at the current graphic subsystem performance.

[polarathene]

Efficient network I/O has already been solved a long time ago with epoll. This patchset (io_uring) delivers efficient asynchronous file I/O, including for buffered file I/O which wasn't possible with the old file AIO interface (io_submit() etc.).

I thought there was quite a bit of discussion about how FreeBSD handles network I/O better than Linux? (no links, or specifics that I can recall though)

[starshipeleven]

I thought there was quite a bit of discussion about how FreeBSD handles network I/O better than Linux? (no links, or specifics that I can recall though)

Lately, all times a certain someone stated it there was someone else posting a benchmark where Linux either trades blows or destroys BSD on networking with 10Gbit cards.

[Farmer]

Problem with your idea its already been tried in the Linux kernel did not provide the performance boost anywhere near expected. Linux kernel is no where near as simple as you think it is.

Snipped for size.

Thank you for taking the time to post that. That was very informative.

[oiaohm]

a short note to a long story: i386 and later have an i/o permission bitmap for supposedly fine grained i/o permission control, also modern hardware is anyway not using classic i/o ports, with (nearly) everything memory mapped ring 3 drivers can drive the hardware just fine without any extra i/o context switching..

https://www.cs.tau.ac.il/~mad/publications/asplos2018-damn.pdf

Theory vs reality. Iommu that you are talking about with it on you are at best down to 90% of the throughput to the compared compared to iommu off. This is without needing to syscall into the kernel to adjust iommu or having kernel call your userspace driver requesting that it give back memory.

Also QNX is quite fast, even was a decade ago, so it is not like it is impossible to do. Also more elegant architecture and algorithms can vastly improve performance, e.g. look at the current graphic subsystem performance.

QNX was not running current day desktop and server workloads.

https://pdfs.semanticscholar.org/93e7/8c3c30982ef3e18bd6fb0fac95976a181de4.pdf

The scheduler things that gave QNX better responsiveness are fairly much mainline Linux now. Please note I said better responsiveness not better performance straight line processing performance..

Remember how I said Linux kernel is hybrid.

https://events.linuxfoundation.org/wp-content/uploads/2017/11/When-eBPF-Meets-FUSE-Improving-Performance-of-User-File-Systems-Ashish-Bijlani-Georgia-Tech.pdf

You see on Linux user space style file systems like micro kernel user put head to head with monolithic style file system driver here. You will also see Linux do this with network drivers as well..

These ring buffers that Linux kernel is getting was one of the tricks QNX and other Microkernels used to keep up historically against monolithic. So the elegant architecture part of Microkernels that reduced context switches Linux kernel is also getting with ring buffers and other zero copy optimisations. Linux kernel is getting more and more of the microkernel elegant methods of reducing context switch and have the monolithic ugly way of reducing context switches and a more modern way with bpf of achieving a lot of the monolithic kernel performance boosts with less security risk.


How were microkernels historically able to keep up and beat historic monolithic kernels..

1) The process to process optimisations microkernels had over monolithic reduced the syscall overhead from application to kernel a lot. This worked out more than the syscalls to the services processing the application request at the time. Problem here is Linux is taking in all of these microkernel gains. So now as a microkernel you don't have syscall saving from application to kernel. Now each context switch between your individual microkernel drivers cost you and put you behind a Linux form of monolithic in performance.
2) Multi threading. Problem is current day Linux kernel is multi threading and NUMA so it is running multi processes.

This is the problem. The tricks microkernels had to claw back performance are no longer unique to Microkernels the Linux kernel uses them as well.

Of course the historic monolithic and microkernels did not have bpf with jit either. This creates a very interest problem. Yes the bpf item breaks the rule of microkernel of all the driver in user-space but if a microkernel like design wants to keep up it will need some for of syscall reduction in kernel space way more complex than just bundling syscalls. This reduction is application puts up request kernel answers from cache with logic if it can resulting in not need to context switch though the driver tree of the microkernel like system of course this is not going to be a microkernel because you will have some of your driver processing in kernel space for performance this in unavoidable. Also when you need to context switch though the userspace driver tree to get stuff done you are going to be slower than the Linux kernel.

Also remember a monolithic kernel can disable like iommu and other security things to just to gain speed. Being a Microkernel you will have to turn these features on just to attempt to get close to the monolithic when it has the security features on. As soon as monolichic starts giving up security for speed there is no way with current design microkernels to keep up.

To make something like a microkernel that can perform against modern day linux kernel will require some very hard choices to optimise it and breaking what it means to be a micro-kernel in places.

[Farmer]

To make something like a microkernel that can perform against modern day linux kernel will require some very hard choices to optimise it and breaking what it means to be a micro-kernel in places.

That's pretty normal though isn't it? Design and implementation always pretty much work that way.

Microkernel: start with a clean sheet. Design a "perfect" model. Render it. Figure out where you're taking a kick in the teeth. Alter that bit, and only that bit, and call it an "optimization."

Monolithic kernel: start with a clean sheet. Design a "perfect" model. Render it. Figure out where you're taking a kick in the teeth. Alter that bit, and only that bit, and call it an "optimization."

As long as you don't muddy the entire thing up you're fine. Work on cleaning up the optimizations.

So the "monolithic" turns into a hybrid - it starts with a clean monolithic design and then gets "optimizations" from the other side of the fence.
So the "microkernel" turns into a hybrid - it starts with a clean microkernel design and then gets "optimizations" from the other side of the fence.

Then a third type, not available when those two were around, enters the arena to replace those two.

Then there are three. All having started from clean models. Received "optimizations" to compete.

Until yet another arrives. Then there will be four.

Rinse and repeated. Endlessly.

How many new programming languages do we have? All designed to be the "one true language" and replace the previous ones.

It always works that way.

Except for COBOL. Tell me that's dead. Please.

[oiaohm]

Microkernel: start with a clean sheet. Design a "perfect" model. Render it. Figure out where you're taking a kick in the teeth. Alter that bit, and only that bit, and call it an "optimization."

Monolithic kernel: start with a clean sheet. Design a "perfect" model. Render it. Figure out where you're taking a kick in the teeth. Alter that bit, and only that bit, and call it an "optimization."

So the "monolithic" turns into a hybrid - it starts with a clean monolithic design and then gets "optimizations" from the other side of the fence.
So the "microkernel" turns into a hybrid - it starts with a clean microkernel design and then gets "optimizations" from the other side of the fence.

That gets you to the start of the Linux kernel.

Then a third type, not available when those two were around, enters the arena to replace those two.

Yes a third type of OS did appear with the SUN Java OS. That is JIT from a verifiable byte-code in kernel space. Linux kernel is getting this as bpf.

Really the Linux kernel coming the cross roads between micro-kernel, monolithic and verified byte-code in kernel space design these days

All having started from clean models. Received "optimizations" to compete.

Linus did not have at the start of Linux the idea that the Linux kernel would be a clean model. Performance was important factor in the path Linus has taken Linux down. The Linux kernel early network stack had drivers in userspace and kernel space this was 100 percent not monolithic or micro-kernel it was the start of the Linux hybrid bitzer nature.


Linux kernel is a true bitzer so when a new model of operating systems appear the developer on Linux look it over and over time bring features of it back into LInux.

This also makes Linux interesting place for comparing the advantages of different OS models because Linux is pure anything but a mixture of the different design models.

Yes calling Linux kernel a monolithic only ignores the kernels history. Early Linux was mostly monolithic key word mostly there were fragments of stuff like microkernel in the network stack. Of course later Linux kernel gets uio and fuse.

https://www.kernel.org/doc/html/v4.18/driver-api/uio-howto.html

Yes proper support for user mode drivers as user space programs. This is like the next generation.

Then we come forwards to bpfilter where you now have kernel loadable .ko modules that are part userspace and kernel space and bpf. Yes part monolithic, part micro-kernel, part verifiable byte-code as one driver to be handled by the kernel. This is a true bitzer.

So Linux just progressive more hybridisation of the models. Network stacks very rarely have been pure monolithic, mirco-kernel or bytecode most have been a bitzer to perform well. Linux kernel is these days seams to be following the path how did network stacks manage to be mostly secure and perform well and apply this to complete OS. Yes there is a 4 model that is not Microkernel, monolithic or byte code verified that is where the Linux kernel seams to be heading and that model as yet does not have a clear name. .

Except for COBOL. Tell me that's dead. Please.

No sorry it not dead companies are still looking for COBOL programmers.

[Farmer]

That gets you to the start of the Linux kernel.



Yes a third type of OS did appear with the SUN Java OS. That is JIT from a verifiable byte-code in kernel space. Linux kernel is getting this as bpf.

Really the Linux kernel coming the cross roads between micro-kernel, monolithic and verified byte-code in kernel space design these days




Linus did not have at the start of Linux the idea that the Linux kernel would be a clean model. Performance was important factor in the path Linus has taken Linux down. The Linux kernel early network stack had drivers in userspace and kernel space this was 100 percent not monolithic or micro-kernel it was the start of the Linux hybrid bitzer nature.


Linux kernel is a true bitzer so when a new model of operating systems appear the developer on Linux look it over and over time bring features of it back into LInux.

This also makes Linux interesting place for comparing the advantages of different OS models because Linux is pure anything but a mixture of the different design models.

Yes calling Linux kernel a monolithic only ignores the kernels history. Early Linux was mostly monolithic key word mostly there were fragments of stuff like microkernel in the network stack. Of course later Linux kernel gets uio and fuse.

https://www.kernel.org/doc/html/v4.18/driver-api/uio-howto.html

Yes proper support for user mode drivers as user space programs. This is like the next generation.

Then we come forwards to bpfilter where you now have kernel loadable .ko modules that are part userspace and kernel space and bpf. Yes part monolithic, part micro-kernel, part verifiable byte-code as one driver to be handled by the kernel. This is a true bitzer.

So Linux just progressive more hybridisation of the models. Network stacks very rarely have been pure monolithic, mirco-kernel or bytecode most have been a bitzer to perform well. Linux kernel is these days seams to be following the path how did network stacks manage to be mostly secure and perform well and apply this to complete OS. Yes there is a 4 model that is not Microkernel, monolithic or byte code verified that is where the Linux kernel seams to be heading and that model as yet does not have a clear name. .


No sorry it not dead companies are still looking for COBOL programmers.

I'd forgotten Sun's OS.

That you for that. It was informative.

[pal666]

AIO also is non blocking and queues data in a buffer, and gives you notification when data hits the disk, useful for disk writes when you need to make sure it got to the disk

i'm pretty sure you are confusing aio with f(data)sync

[Happy Heyoka]

Except for COBOL. Tell me that's dead. Please.

ah no. somewhere out there is a person that makes more in a day maintaining 50 year old Cobol than we make in a month combined.
They're having a really shit time doing it, but they own three houses.