READFILE System Call Rebased For More Efficient Reading Of Small Files

[oiaohm]

You are kidding me, optimizing one atomic add away? Lol

Atomic locking of ulimit, even my i486 and Sgi Octane are still laughing about that one. Man that was a good one. lol ;-)

OMG no! Thanks god you are not kernel developer ;-) But see how flexible a simple vsyscall for any such syscall would be?

Really with this answer I can understand why your idea is being rejected by kernel developers. The overhead of atomic operation overhead increases as your core count goes up. A i486 that a single core or a Sgi Octane that at max is only 2 core is not going to have much of a atomic operation overhead.

Yes you got it wrong its 2 atomic operations removed. 1 add that is the open and 1 subtract that is the close. I did write both. So readfile takes away 2 syscalls and two atomic operations.

While optimizing a ulimit addition is already laughable, many probably consider it correct operation if your ulimit is 256 and your 257'th readfile() would fail as an open would.

With the atomic memory operations this has very little overhead. Its your value changing that causes overhead with atomic operations. So a ulimit of 256 and you are attempting 257 file this could get interesting. readfile could technically be allowed to proceed because its not increase number of actively open files. This kind of allowance is not possible with the vectored syscall allowance.

Think for reading system setting values from sysfs or procfs does it really make sense to be increasing ulimit count if you are not keeping those files open for some reason.

See how vsyscall for any sequence of open, write, close would be handy, too? ;-)

Absolute not a much as a proper writefile command. This is because you have ignored the overhead of atomic operations. Yes your response show you did not understand that a atomic operation cost on a modern 64 core 128 thread cpu is way worse than i486 that is a single thread or a Sgi Octane that at best is 2 threads.

rene you are being absolute clueless. Vectored syscall idea has major limitations. A direct syscall for readfile and writefile makes more sense today than it did in past due to the increasing number of cores resulting in the increased cost of locking and atomic operations by having more core/threads to keep on the same page.

Think about it you have 128 threads in a process all doing a readfile operation removing ulimit alteration from that means those operations can be processed inside each thread of the cpu without having to check with any other core if doing this is fine.

Writefile does get trick again because you do have to have a atomic operation for altering a value in procfs and sysfs but this is from 3 atomic operations to 1.

[rene]

This is still you not understanding the problem. Readfile syscall does not just gain because 3 syscalls come one. Remember a normal open syscall has to add 1 to the ulimit file count and close subtracts 1 the Readfile syscall can skip this. Remember current ulimit values have to be shared so changing them can be quite performance costly.

You are kidding me, optimizing one atomic add away? Lol

Small file reads are in fact used a lot even in a sysvinit. You think PIDfile checking to prevent starting a service twice. There are quite a few small file usage presure points once you go looking for them. These are all hidden in utility programs that are used over and over again. Systemd with more complex cgroup process management is reading more small files. The effects of ulimit changes having to be shared also gets bigger. Think systemd is going to be used by KDE and Gnome around every started application.

The reality here is a straight up vectored syscall will not deal with the ulimit issue caused by open and close because you have it performing a normal syscall. A vectored syscall to properly handle error events is going to require setup so you cannot be sure u_ioring will be any worse than doing a vectored syscall due to how heavy doing proper error handing is going to be.

While optimizing a ulimit addition is already laughable, many probably consider it correct operation if your ulimit is 256 and your 257'th readfile() would fail as an open would.

Also error handling or a vectored syscall for the normal success case is as simple as for one read or open: it it returned success all vectored calls succeeded. Even for the error case it is not much harder, and could be abstracted in a libc readfile shim.

Yes readfile vs a vector syscall doing (open,read,close) the readfile is going to be faster because the vectored is going to have allow for the possibility it does not contain a close so change ulimit on open or close or process what vectored syscall has been passed to attempt to work out if it does not have to do ulimit change.

Vectored syscall is more flex able but it that flexibility that comes back and kicks you where it hurts with the open read close problem. Dealing with that flexibility issues does come with its own overhead problem.

Really I would like to see a writefile as well.

See how vsyscall for any sequence of open, write, close would be handy, too? ;-)

Those asking for a readfile syscall in Linux have correctly looked at the problem. Yes just like indepe said if you wanted vectored syscalls having readfile syscall is not mutually exclusive. You have failed to notice the all advantages the readfile syscall gets its reduction in syscalls it also reduction in atomic locking on ulimit a valve due to not needing to increase and decrease the file open count. Yes this is another bit of error handling how to handle the problem of max open file count.

Atomic locking of ulimit, even my i486 and Sgi Octane are still laughing about that one. Man that was a good one. lol ;-)

For a everything is a file operating system having a open read close and open write close as single syscalls has their place.

OMG no! Thanks god you are not kernel developer ;-) But see how flexible a simple vsyscall for any such syscall would be?

[rene]

Depends on which built-in. But, the speed of shell scripts is typically limited by using non-builtin commands, which is why I'm concerned with how quickly they execute.

I have news for you: readfile() will not affect the speed of which shell scripts and especially non-builtin commands execute at all.

[indepe]

Some of us have mentioned using readfile() via io_uring, which is the same basic idea and likely to be possible as soon as readfile() is merged in.

Certainly.

Also, in any case, regarding passing on file descriptors in io_uring:

If using the IOSQE_FIXED_FILE flag on IORING_OP_OPENAT, the addr2 field could be used as an "index into the files array registered with the io_uring instance". That might even generally be an easier way to initialize and/or update the registered files array. So it would just use existing features and data structures, just by adding that flag/option to the open file operation.

However the application would need to maintain the indexes, so in our use case readfile as its own operation would be much easier.

[coder]

The two ideas are not mutually exclusive. Even if there was a vectored syscall, I would still like to use readfile() inside that vectored syscall.

So if utility or app wants to read thousands of sysfs values, it can issue a single vectored call with thousands of readfile() commands. That seems easier to use, avoids any mental acrobatic about passing on the file descriptor, and combines efficiency improvements of both.

Some of us have mentioned using readfile() via io_uring, which is the same basic idea and likely to be possible as soon as readfile() is merged in.

[coder]

How many small file do shell built-in open?

Depends on which built-in. But, the speed of shell scripts is typically limited by using non-builtin commands, which is why I'm concerned with how quickly they execute.

[coder]

That would be nice. Of course they chose GPL incompatible library for this.

The point isn't about any one implementation or even GCD, itself, but rather the mere fact of having kernel support for work-stealing.

Btw. isn't OpenMP alternative to it?

libgomp, specifically, is a piece of garbage, in my opinion. It uses userspace spinlocks, by default (though some distros disable this). And the OpenMP standard just focuses on the user interface for exposing concurrency, whereas my concern is about having the necessary platform support for efficient, scalable, and well-behaved implementations of it (and other APIs, too).

[oiaohm]

Yeah, since 20+ years I contribute and maintain a source distribution (t2, previously known as ROCK Linux) scripted in Shell, but I don't know shell. Again, this few system calls saved is still not eve a percentile of the CPU cycles spend. Also nice try throwing large numbers like 90% when the total speedup to gain if any is 0.00001%. So yeah, port some 20 applications and get total speedup of 0,000009%. Maybe. Maybe the additional setup overhead and also contribute to a 0.00001% slowdown as again this small file access is nothing in the total grand scheme of boot time you are talking about. have a good day.

This is still you not understanding the problem. Readfile syscall does not just gain because 3 syscalls come one. Remember a normal open syscall has to add 1 to the ulimit file count and close subtracts 1 the Readfile syscall can skip this. Remember current ulimit values have to be shared so changing them can be quite performance costly.

Small file reads are in fact used a lot even in a sysvinit. You think PIDfile checking to prevent starting a service twice. There are quite a few small file usage presure points once you go looking for them. These are all hidden in utility programs that are used over and over again. Systemd with more complex cgroup process management is reading more small files. The effects of ulimit changes having to be shared also gets bigger. Think systemd is going to be used by KDE and Gnome around every started application.

The reality here is a straight up vectored syscall will not deal with the ulimit issue caused by open and close because you have it performing a normal syscall. A vectored syscall to properly handle error events is going to require setup so you cannot be sure u_ioring will be any worse than doing a vectored syscall due to how heavy doing proper error handing is going to be.

Yes readfile vs a vector syscall doing (open,read,close) the readfile is going to be faster because the vectored is going to have allow for the possibility it does not contain a close so change ulimit on open or close or process what vectored syscall has been passed to attempt to work out if it does not have to do ulimit change.

Vectored syscall is more flex able but it that flexibility that comes back and kicks you where it hurts with the open read close problem. Dealing with that flexibility issues does come with its own overhead problem.

Really I would like to see a writefile as well.

Those asking for a readfile syscall in Linux have correctly looked at the problem. Yes just like indepe said if you wanted vectored syscalls having readfile syscall is not mutually exclusive. You have failed to notice the all advantages the readfile syscall gets its reduction in syscalls it also reduction in atomic locking on ulimit a valve due to not needing to increase and decrease the file open count. Yes this is another bit of error handling how to handle the problem of max open file count.

For a everything is a file operating system having a open read close and open write close as single syscalls has their place.

[indepe]

The two ideas are not mutually exclusive. Even if there was a vectored syscall, I would still like to use readfile() inside that vectored syscall.

So if utility or app wants to read thousands of sysfs values, it can issue a single vectored call with thousands of readfile() commands. That seems easier to use, avoids any mental acrobatic about passing on the file descriptor, and combines efficiency improvements of both.

Instead of all of this speculation, why don't you just profile and debug, and figure out where and why the bottlenecks occur, so that you can actually fix them?