Announcement

Collapse
No announcement yet.

READFILE System Call Rebased For More Efficient Reading Of Small Files

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by rene View Post
    sorta, however, io_uring is way more complex to setup,
    So, maybe propose a userspace convenience function that simplifies usage for what you want.

    Originally posted by rene View Post
    the last time I checked does not directly support this open, read, close sequence,
    If we have readfile(), then it doesn't have to.

    After io_uring and readfile(), are there any problems left to be solved by your vectorized system call? Maybe you just need to move on and find a new battle to fight.

    IMO, readfile() seems like a weird thing to get so worked up over. It solves a narrow problem nicely, conveniently, and safely, and should be usable with io_uring for added benefit.

    Comment


    • #12
      Originally posted by coder View Post
      IMO, readfile() seems like a weird thing to get so worked up over. It solves a narrow problem nicely, conveniently, and safely, and should be usable with io_uring for added benefit.
      Yeah, as a desktop app dev I always wondered why there's no glibc/syscall to just read the damn file without having to go thru 15 error checks at different stages, setting up a buffer, special heuristics for abstract Linux kernel files which report fake file sizes, and whatnot.
      I'm glad it's being abstracted away by a simple syscall, not to mention it'll be faster too.

      Comment


      • #13
        Originally posted by rene View Post
        However, all of my proposals would just work with your examples. Of course they would return which syscall were run and their respective error codes. So depending on which proposal you look at, the user space app would either get to know read() failed and close() was not executed and thus do it if needed,
        This has missed what readfile is doing. Readfile the userspace does not need to execute extra code to close anything. To provide a vectored syscall that can replicate readfile you need to provide smarter error handing. As in the open succeeded so the close has to be run even if the read fails.

        To get a more universal in have to deal with the error problem. Remember part of the readfile syscall objective is to make userspace code less error prone and shorter and less syscalls.

        Readfile in worst case is 1 syscall. Readfile done by the vector you proposed that the user space app would know it failed its now 2 syscalls. When your objective is to reduce syscalls this does mean your vectored syscall idea has to deal with the errors in a sane way so that the userspace is not requiring to syscall to fix up after vectored syscall. This is where the proposals of vectored syscalls have fallen down.

        Comment


        • #14
          Originally posted by coder View Post
          So, maybe propose a userspace convenience function that simplifies usage for what you want.


          If we have readfile(), then it doesn't have to.

          After io_uring and readfile(), are there any problems left to be solved by your vectorized system call? Maybe you just need to move on and find a new battle to fight.

          IMO, readfile() seems like a weird thing to get so worked up over. It solves a narrow problem nicely, conveniently, and safely, and should be usable with io_uring for added benefit.
          readfile() is a solution for an unimportant niche use case nobody asked for. This should be solved more generically. Vectored system all would be useful for a wide class of performance optimizations, and io_uring should be better adapted to solve the "continue with previously fd handle result" than this clutch or their "set fixed fd for open" hack they proposed for supporting this syscall sequence in io_uring, ...

          Comment


          • #15
            Originally posted by cl333r View Post

            Yeah, as a desktop app dev I always wondered why there's no glibc/syscall to just read the damn file without having to go thru 15 error checks at different stages, setting up a buffer, special heuristics for abstract Linux kernel files which report fake file sizes, and whatnot.
            I'm glad it's being abstracted away by a simple syscall, not to mention it'll be faster too.
            reading a file is not 15 error checks, readfile() replaces exactly 3 with one, open(), read(), close() and is meant for small files only. You sill need to bring your own buffer. So getting exited about this is a bit interesting, if you are lazy you also only need to check one or two of the classic open(), read(), close sequence, namely of read(), or if you don't want to be that lazy better open, too ;-) So it's not like readfile() will revolutionize programming.

            Comment


            • #16
              Originally posted by oiaohm View Post

              This has missed what readfile is doing. Readfile the userspace does not need to execute extra code to close anything. To provide a vectored syscall that can replicate readfile you need to provide smarter error handing. As in the open succeeded so the close has to be run even if the read fails.

              To get a more universal in have to deal with the error problem. Remember part of the readfile syscall objective is to make userspace code less error prone and shorter and less syscalls.

              Readfile in worst case is 1 syscall. Readfile done by the vector you proposed that the user space app would know it failed its now 2 syscalls. When your objective is to reduce syscalls this does mean your vectored syscall idea has to deal with the errors in a sane way so that the userspace is not requiring to syscall to fix up after vectored syscall. This is where the proposals of vectored syscalls have fallen down.
              I have not missed what this tiny little readfile() should be doing. I proposed smart error handling earlier, beside error handling in an open(), read(), close() sequence is not exactly rocket science and I repeatedly summarized my two proposals for that in the previous post. the minor complexity of setting up a 3 vector syscall could be easily wrapped in a (g)libc readfile() like we usually do and my proposed vsyscall is also only one syscall not two as you claim for this or any other sequence. The close() fixup is also only needed for one of my two proposals, and even therein that only in an error case.

              However, the proposal is useful for actual real world optimizations and problems, starting with using multiple reads as in open, read, read, read, close to save data in different locations, or mix it with readv or use it for write or any other more interesting syscall, too. That is os innovation, and not quick readfile() hacks that nobody even freaking TM asked for. For a niche performance optimization of maybe mostly system proofing tools.

              In any case, it is apparently pointless to discuss OS engineering in this new site forums, I have better things to do ;-)

              Comment


              • #17
                Originally posted by rene View Post
                readfile() is a solution for an unimportant niche use case nobody asked for.
                This is nonsense. Try running some common shell commands via strace, sometime. They often read several configuration files and potentially some things in sysfs, as mentioned. Especially if little config files are cached, then you get real benefit by optimizing them down to a single syscall.

                Over the years, I've often been struck by the relative lack of improvement in shell script execution speed. Sure, it's gotten faster, but not at the same rate as either CPUs or storage. And I think this goes directly to helping such cases.

                Originally posted by rene View Post
                io_uring should be better adapted to solve the "continue with previously fd handle result" than this clutch or their "set fixed fd for open" hack they proposed for supporting this syscall sequence in io_uring, ...
                Your desperation is clearly showing. You still attack the io_uring alternative to readfile(), even though readfile() is happening.

                Originally posted by rene View Post
                open, read, read, read, close to save data in different locations
                And you talk about niche optimizations? The cases where that's actually faster than just copying the data in userspace are indeed small. And one doesn't usually know the locations and/or sizes of any further substantial reads until at least the file header is read.

                Furthermore, where there are enough reads of sufficient size for this to be a real win, it should be easy to justify the setup overhead of io_uring.

                Originally posted by rene View Post
                That is os innovation, and not quick readfile() hacks
                This is Linux, not some nano-kernel. Linux values pragmatism over minimalism.

                Originally posted by rene View Post
                In any case, it is apparently pointless to discuss OS engineering in this new site forums
                Yes, clearly the reason why we don't agree is that we're just plebs on some news site, incapable of appreciating the true elegance of your vision.

                Originally posted by rene View Post
                I have better things to do
                Well, you brought it up! So, I'd have to say apparently not.

                Comment


                • #18
                  Originally posted by rene View Post

                  reading a file is not 15 error checks, (bla bla)
                  It was obviously an exaggeration, you should know it.

                  Comment


                  • #19
                    Originally posted by coder View Post
                    This is nonsense. Try running some common shell commands via strace, sometime. They often read several configuration files and potentially some things in sysfs, as mentioned. Especially if little config files are cached, then you get real benefit by optimizing them down to a single syscall.

                    Over the years, I've often been struck by the relative lack of improvement in shell script execution speed. Sure, it's gotten faster, but not at the same rate as either CPUs or storage. And I think this goes directly to helping such cases.

                    ...
                    that you believe readfile() can in any way, shape or form speed up shell script execution speed is laughable at best as reading (small) files is not a bottleneck there and does not even register in a profile. I skip over commenting all the other non technical personal attacks.

                    This has nothing todo with nano kernel but reasonable OS design, maybe ask the BSD folks or Apple's macOS what they think about this Posix extension ;-)
                    Last edited by rene; 04 April 2021, 10:05 AM.

                    Comment


                    • #20
                      Originally posted by rene View Post
                      I have not missed what this tiny little readfile() should be doing. I proposed smart error handling earlier, beside error handling in an open(), read(), close() sequence is not exactly rocket science and I repeatedly summarized my two proposals for that in the previous post. the minor complexity of setting up a 3 vector syscall could be easily wrapped in a (g)libc readfile() like we usually do and my proposed vsyscall is also only one syscall not two as you claim for this or any other sequence. The close() fixup is also only needed for one of my two proposals, and even therein that only in an error case.
                      Why does close file accept null these days that right because of Reality. Reality is developer are great at messing up error handling this happened with close failing to include if(null) before performing close resulting in bugs. Saying needs to be wrapping inside glibc is admit you have screwed up as well the Linux kernel does not just support 1 version of libc there are many more compact than glibc used in embedded solutions the Linux kernel is used in. The one that need close() fixup need to be rejected straight up because it will cause errors like file close when it would not accept a null out file handle.

                      Other problem the other example where it executes forwards anyhow thinking the virtual syscall is to take any syscall you could run into a old style close file problem with particular syscalls where if the open fails where the close should not run. There are some more funny ones where the read fails and the file auto closes and returns error.

                      The reality is there need to be some form of logic with the multi listed syscalls that virtual syscall suggestions kind of like some form of execution metadata as in if A syscall works Z syscall must be run. Remember the suggested vectored syscall thing did not say stop at 3 syscalls. Of course this gets more complex when you get into syscalls where different error values equal different responses.

                      Originally posted by rene View Post
                      However, the proposal is useful for actual real world optimizations and problems, starting with using multiple reads as in open, read, read, read, close to save data in different locations, or mix it with readv or use it for write or any other more interesting syscall, too. That is os innovation, and not quick readfile() hacks that nobody even freaking TM asked for. For a niche performance optimization of maybe mostly system proofing tools.
                      O by that gets more complex. open read(security error) read close. Fun point that second read just got your program terminated. There are more issues than just the syscall error return value. Remember things like "segmentation fault" a open file may be a mmap if you read past end hello segmentation fault or there is a section of the mmap you don't have permissions to access again "segmentation fault". These are raised exceptions.

                      Remember some of these raised exceptions if they happen again before exception handler processes the exception the Linux kernel response that is valid is process termination. When I say you need to deal with the error handling problem this is the error value from the syscall and what ever exceptions the syscall causes to be set particularly once you start stacking syscalls one after the other.

                      Vectored syscalls sounds like a simple easy idea until you wake up the error handling cannot be half assed. Yes the error handling so doing multi syscalls in one syscall does not result in application termination is not simple. Remember while you are in a syscall the user space exception handler is not being run its run when the syscall ends.

                      I guess when I was saying error handling you were thinking only syscall error values not the raised exception errors.

                      I have not seen any vectored syscall proposal that has done the error handling good enough not to either result in items left open that should be closed or being a trigger to application termination due to either not handling syscall error values or not handling exceptions.

                      Yes this would result in vector syscall lookling like vsyscall(some error handling structure, [syscall], [syscall]....)

                      That error handling structure from every time I have looked at this problem end up horrible complex and end up basically with a bytecode engine in the kernel and the kernel already has a byte code engine in bpf that what leads to lets extend bpf to ebpf that comes Linux bpf today.

                      rene the problem looks simple until you stop avoiding the error handling problems. Yes io_uring working with bpf is to deal with some of the problem cases syscalls as well where the bpf code in kernel can be smart on don't do anything that will cause process termination..

                      Comment

                      Working...
                      X