Newest Linux Optimizations Can Achieve 10M IOPS Per-Core With IO_uring

[Danny3]

Not only do they need hardware, look at this page https://www.argyllcms.com/downloadsrc.html ... it's obvious the graphics guys would need a git powered hosting service as well. Just look at that graphical design. Zip archives as the main distribution medium. Eh wtf.

That makes me even more angry at the Linux foundation lack of action.
I wonder who's running it, because it looks like old corrupt politicians.

[stormcrow]

Does Jens need actual disks to test maximum throughput? You don't want the disk to bottleneck your measurements I'd think, so likely he uses a tmpfs ramdisk or some other means to determine upperbound I'd guess.

Yeah, you do need actual hardware. It's the hardware that defines what the bounds and conditions will be. It doesn't do a whole lot of good to play around with RAM based filesystems only to find out the real hardware has practical considerations you didn't know about and aren't in the specs and completely trash your assumptions.

[coder]

Yeah, you do need actual hardware. It's the hardware that defines what the bounds and conditions will be. It doesn't do a whole lot of good to play around with RAM based filesystems only to find out the real hardware has practical considerations you didn't know about and aren't in the specs and completely trash your assumptions.

Yeah, not only that, but he's presumably working on optimizing it for deployment on real hardware -- not just benchmark bragging rights. So, you want to make sure you're measuring the full stack that you're going to use in production. Otherwise, there could be bottlenecks that he's not seeing, and therefore not optimizing.

[coder]

That makes me even more angry at the Linux foundation lack of action.

Yeah, we wish Linux kernel developers would get free hardware, but it's not clear to me how you expect this to work in practice.

Do you imagine someone decides to start working on some kernel feature and then asks the Linux Foundation to send them some free hardware? At this point, how do we know if their code is any good, or if they'll even finish? Would companies donate all the additional money needed just to buy free hardware for a bunch of randos to work on features that might or might not be useful or ever get finished?

Most skilled kernel developers seems to land employment somewhere that will sponsor their development. Then, the employer pays Linux Foundation for infrastructure & admin to run the site, regression tests, pay Linus' bills, etc. All the commons stuff they depend on that's not done by their employees or contractors. WTF does some server vendor care if Linux has good HDR support or not?

I wonder who's running it, because it looks like old corrupt politicians.

The irony is that what you seem to want is a lot more like government grants than the quasi service-oriented model of how Linux Foundation actually works. Companies voluntarily contribute, because they need Linus be available & have the resources needed to merge in & test their changes. Not because they like giving to charities or want to subsidize a bunch of crusty, rando kernel hacker wannabe basement-dwellers.

It's not like what governments do, by taxing everyone and then redistributing those resources.

But, I mean, sure. Let's forget everything good Linux Foundation has done and start blaming it for everything Linux isn't. That's certainly a lot cheaper & easier than donating your own money to someone working on a feature you want, or actually rolling up your sleeves and make your own contributions!

[NobodyXu]

Not only do they need hardware, look at this page https://www.argyllcms.com/downloadsrc.html ... it's obvious the graphics guys would need a git powered hosting service as well. Just look at that graphical design. Zip archives as the main distribution medium. Eh wtf.

There’re lots of free git hosting services around, like github, bitbucket, etc.

Zip archives are provided to make downloading the software easier.

[sdack]

Yeah, you do need actual hardware. It's the hardware that defines what the bounds and conditions will be. ...

No. There is no technical reason why one could not just use a software or dummy device. We already have these for various reasons in the kernel. He is simply getting paid to do it and so it is a matter of professionalism. People do not only want theoretical numbers, but they want to know how much they get out of it in a real-world scenario and this is best demonstrated with available hardware.

I hope he does not do it the way you have described it. I would hate to see the block layer getting tweaked for one particular device, with the real chance of making it worse for other devices, because of assumptions based on a specific Intel Optane SSD.

[coder]

No. There is no technical reason why one could not just use a software or dummy device. We already have these for various reasons in the kernel.

Okay, so let's say he does that, gets some great numbers, and then his employer runs on actual hardware and encounters a bottleneck in the NVMe driver? It could even be the case that there's some nonlinear interaction between the two, such as thrashing the CPU's instruction cache, because the whole stack won't fit in it at once. Those are the sort of reasons you can't simply do all of the tuning in isolation.

When what you care about is actual quarter-mile times, you can't do all your testing & tuning without even putting the engine in the car. What happens if a diff or CV joint can't handle the torque with the additional strain caused by wheel-hopping? How would you decide the amount of downforce needed vs. simply adjusting the weight distribution? What if a bigger turbo produced better numbers on the dyno, but yields a torque band too narrow for the transmission?

I hope he does not do it the way you have described it. I would hate to see the block layer getting tweaked for one particular device, with the real chance of making it worse for other devices, because of assumptions based on a specific Intel Optane SSD.

It sounds like he is making block-layer changes, but of course there are reviewers and everyone is free to run benchmarks on other device types. Most likely, the things he's doing to reduce overhead for his case will also benefit everyone else. It's when you get into things like I/O schedulers that advantaging one class of device can really hurt another.

[GrayShade]

No. There is no technical reason why one could not just use a software or dummy device. We already have these for various reasons in the kernel.

You can't optimize the code you're not testing.

[partcyborg]

No. There is no technical reason why one could not just use a software or dummy device. We already have these for various reasons in the kernel. He is simply getting paid to do it and so it is a matter of professionalism. People do not only want theoretical numbers, but they want to know how much they get out of it in a real-world scenario and this is best demonstrated with available hardware.

I hope he does not do it the way you have described it. I would hate to see the block layer getting tweaked for one particular device, with the real chance of making it worse for other devices, because of assumptions based on a specific Intel Optane SSD.

Then you should have no problem telling me how to create an in-memory nvme block device. Not a loopback or tmpfs volume, but a bona-fide /dev/nvme* device managed by the nvme kernel driver but not actually attached to physical media.

Go ahead, I'll wait 🙄

[cl333r]

While I know that io_uring will only make a difference on servers (a.k.a. systems servicing lots of requests) and therefore won't see a difference on a desktop, it seems his solution is so fast that one day we'll find out there's something fundamentally wrong with it.