GNU Coreutils 9.5 Can Yield 10~20% Throughput Boost For cp, mv & cat Commands

[phoronix]

Phoronix: GNU Coreutils 9.5 Change Can Yield 10~20% Throughput Boost For cp, mv & cat Commands

While the uutils Rust-written Coreutils effort has been chugging along, the upstream GNU Coreutils effort is showing no signs of slowing down. Out today is GNU Coreutils 9.5 with yet more feature work and bug fixes including a security fix for a chmod issue that's been around since the beginning...

https://www.phoronix.com/news/GNU-Coreutils-9.5-Released

[bug77]

Nice screen-cap. So the change was in the comments?

[varikonniemi]

If increasing the minimum gives better performance, why has it not used higher values previously and minimum only when needed?

Also, does that chmod issue mean that any user has been able to change and execute any file since the beginning of time?

[sophisticles]

If increasing the minimum gives better performance, why has it not used higher values previously and minimum only when needed?

I was literally about to ask the same question.

I could understand if they changed the maximum value but they changed the minimum value.

They could have just used 256KiB all along.

In fact based on the comments in the code, one could conclude that perhaps using a minimum value of 512KiB might be even better.

Also, does that chmod issue mean that any user has been able to change and execute any file since the beginning of time?

From the announcement:

chmod -R now avoids a race where an attacker may replace a traversed file
with a symlink, causing chmod to operate on an unintended file.
[This bug was present in "the beginning".]

If "the beginning" refers to when coreutils first came into being, then that would be 2002, if it refers to when fileutils was created, then that would be 1990.

What drives me up the wall is that the announcement does make it sound like they have known about this all along and only now, after 34/22 years they decided to fix it.

Alternatively, it could be that they only recently leaned about it, but that still doesn't look good that such a bug could exist for so long undetected.

I have been saying for decades that the supposed superiority of open source software is a myth, a scam, promoted by people with ulterior motives and believed by people that lack the technical knowledge to analyze the claims properly.

Then you have people like me, that know the truth and use it sometimes primarily to avoid having to pay per seat licensing fees associated with properly coded software.

[byteabit]

A little bit competition is healthy.

[yoshi314]

In fact based on the comments in the code, one could conclude that perhaps using a minimum value of 512KiB might be even better.

those kinds of assumptions tend to fall apart when the hardware situation changes. some new piece of hardware comes along and upends the assumptions. or the code was not tested on certain unusual configuration (e.g. a big fileserver). maybe the fs has nonstandard block size.

Sometimes it's better to be a bit more conservative, unless someone rolls some decent benchmarks.


it reminds me of one notable example of optimistic assumptions.

back in the day linux kernel would keep a dirty bytes queue of (up to) 10% of system ram. it was used for caching that was being written out to storage devices, not to keep user's program waiting - it would just go on in the background once all remaining data was in the dirty buffer.

That limit made sense when your machine had < 64MB of ram. Stopped making sense when 64bit came along and systems with 4GB+ of ram became the norm. But nobody remembered about it.

This was the cause for major slowdowns on 64bit installations when writing a lot of data to slow usb 2.0 devices. I am not exactly sure why 32bit systems did not suffer from this issue, or at least not to the extent of 64bit systems.

When writing a ton of data (let's say 500MB), the kernel would queue up to 10% of available ram (so potentially up to ~400MB on 4GB of ram), and start flushing it out to slow usb storage device. It blocked the entire system, because at the time there was one shared queue for dirty bytes for all devices. If you queued up 300MB of writes to usb, they were handled in the order they came in intertwined with other i/o operations.

once you queued up the write - your desktop would grind to a halt. Once the usb device data was flushed - entire system suddenly sprang back to life.

I kid you not, it took YEARS to narrow down and fix. i've seen people going back and forth about it on various forums, proposing various fixes. per device queues and more sane defaults for that queue size finally nailed that down, among other fixes that linux picked up along the way.

[wagaf]

Then you have people like me, that know the truth and use it sometimes primarily to avoid having to pay per seat licensing fees associated with properly coded software.

You mostly show your deep bias and ignorance.

There is a lot of shit proprietary code, and there is shit open source code too.

Microsoft and Apple systems also have legacy, shit proprietary pieces of code all around. And they also use Open Source software all around.

But chmod is not shit. It's just that not a single piece of code is perfect.

[Akiko]

If increasing the minimum gives better performance, why has it not used higher values previously and minimum only when needed?

As a kernel dev I can give you a hint:
1. cache sizes: Trying to stay below the size of the first and second level cache yields better results in the hw prefetchers and usually better performance. (To big buffer may result in cache trashing.)
2. different platforms: You try to go for values which work well on most of the platforms, especially in embedded devices.
3. page sizes (goes hand in hand with 1 and 2): there is a direct connection between page sizes, I/O and memory allocation. 4k pages where quiet common for a while, but there are also 16k, 64k, 2m and 1g page sizes. Though, I must admit that it depends highly and on the implementation details which leads to the 4. issue, which is the most important one also depends on 1, 2 and 3.
4. I/O sizes and schedulers: the older smaller values did work very well with the sectors sizes of harddisks (512b and 4k), but now flash memory is a thing, buffered by various amounts of local DRAM. Though, flash is also done in sectors, 64k were quite common and shifted to 128k sectors with bigger NAND/NOR flashes (usb sticks). And todays NVMe harddrives seem to got with 256k sectors. On Linux you can actually test this with some simple methods like "dd bs=128k/256k ..." and you may see a sweet spot. The Linux schedulers always try to collect several pages before working on them, on weak hardware that can result in I/O stalls. If you want to see it yourself, in linux/mm/page_alloc.c (grep for zone_managed_pages) are some of these critical paths which can lead to some ugly stalls on systems which have a low core count or just a weak cpu, or especially worse, a single cores and very fast storage memory like NVMe.

If you wonder why such simple tools for copying files can be such a trouble: well they are just some nice wrappers around syscalls, and hitting on syscalls in a high frequency is just another problem topic.

[rene]

updated: http://t2sde.org/packages/coreutils

[sophisticles]

But chmod is not shit. It's just that not a single piece of code is perfect.

Then how do you explain a security vulnerability that has existed for 34 years?

But I am the biased, ignorant troll.

LOL!