Announcement

Collapse
No announcement yet.

New readfile() System Call Under Review For Reading Small~Medium Files Faster

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by jaskij View Post
    atomsymbol I've run precisely into this sysfs bottleneck when doing fast switching of GPIOs in an industrial PC. I ended up keeping those fds open and doing seek()/write() over open()/write()/close() - it cut down my CPU utilization by 90%.

    ​​​​
    Sorry. But that's just stupid. Doing high volume GPIO-switching from userspace using sysfs...
    I see these tendencies all over the place. "I have installed a Raspberry Pi. I know embedded!".
    Sysfs was never meant as a high speed/volume interface and this does not change that.
    It just reduces the amount of syscalls from userspace.
    The kernel still has to do vfs-crap and open/seek/read/close for every state.
    And that's for something that should be a byte memory access read away in a CPU GPIO bank register.

    Comment


    • #12
      Can anyone tell me the maximum filesize for this syscall?
      Yes, i did read the code, and the new manpage. Nothing seems to indicate what qualifies as "small". The post also mentions "small and medium sizes".

      It even has testcases:

      + test_filesize(0x10);
      + test_filesize(0x100);
      + test_filesize(0x1000);
      + test_filesize(0x10000);
      + test_filesize(0x100000);
      + test_filesize(0x1000000);

      Where that largest testcase is 16 MiB.

      I'm asking because i want to know if this syscall can be used to, for example, load icons and config files. Now config files are likely sub megabyte ones. But icons can be multiple megabytes (think about ico files that contain multiple images). But besides that, think about reading thumbnails for display in, say, dolphin or gwenview etc...

      Now i'm not betting on this to be "substantially faster" then 3 syscalls (open/read/close) as the syscalls themselves are probably just a tiny percentage of actually handling the file data to do something with it (say jpeg/png/svg/webp decoder). But still, if you expand this over large folders it might become a notable difference.

      Lastly, and this one is specifically for phoronix. Greg is, in the next version of this patch, going to post benchmarks too: https://lore.kernel.org/lkml/[email protected]/

      Comment


      • #13
        Originally posted by jaskij View Post
        atomsymbol I've run precisely into this sysfs bottleneck when doing fast switching of GPIOs in an industrial PC. I ended up keeping those fds open and doing seek()/write() over open()/write()/close() - it cut down my CPU utilization by 90%.

        ​​​​
        You are abusing the platform. Industrial + raspberry pi does not mix well.
        You should be using a RTOS platform. Even an ESP8266 would be better suited in your case.

        Comment


        • #14
          markg85 whoever said I was using a Pi? It was an i.MX 5 SoM. And yes, fast switching was kind of stupid, but our client wanted it so what was I to do? Also, on a single core 500 MHz Cortex-A7 I'll take all the savings I can get.

          Comment


          • #15
            Originally posted by jaskij View Post
            markg85 whoever said I was using a Pi? It was an i.MX 5 SoM. And yes, fast switching was kind of stupid, but our client wanted it so what was I to do? Also, on a single core 500 MHz Cortex-A7 I'll take all the savings I can get.
            Oops, i messed up there. I read @milkylainen post first (which mentions the raspberry pi and was a reply to yours) and blindly assumed that you were using a raspberry pi. I did read your post but just assumed the wrong thing. Sorry for that.

            I stand by my suggestion that you should probably need to go for an RTOS system. Also, who knows better? You, the developer, or your client? I'm going to assume (again) that you know better and should convince your client of the better approach.

            Comment


            • #16
              markg85 nowadays I would. But I was a noob who got thrown into a middle of a project. The hardware was done, the client had stupid requirements and I was a noob trying to make the best out of it.

              These days I'd probably tell them to just slap a hardware PWM or an MCU in there. But if you're doing small runs anything software-defined is usually not worth the added R&D costs if you can pay extra 5$/device in hardware.

              Also, I stand by my position that for cheap stuff with a 500 MHz Cortex-A7 I'll take all the savings I can get.

              P.S.
              Sorry for not quoting, Phoronix forums are iffy on mobile.

              Comment


              • #17
                Originally posted by atomsymbol View Post
                Is there a real-world scenario in which open/read/close of sysfs and procfs files is a bottleneck?
                Yes, there is.

                Comment


                • #18
                  Originally posted by atomsymbol View Post
                  Without the ability to read multiple small files using a single system call, it is impossible to increase IOPS (unless using multiple threads).
                  IOPS is Io Operations Per Second. You can increase them either by overlapping more operations or decreasing the time between successive operations. This addresses only the latter, though still improves performance (which, again, is their reason for doing it)

                  If you also want the former, then you can simply use it with io_uring to queue up multiple readfile() operations.

                  So, you can have your cake and eat it!

                  Comment


                  • #19
                    Originally posted by markg85 View Post
                    Can anyone tell me the maximum filesize for this syscall?
                    Yes, i did read the code, and the new manpage. Nothing seems to indicate what qualifies as "small". The post also mentions "small and medium sizes".

                    It even has testcases:

                    + test_filesize(0x10);
                    + test_filesize(0x100);
                    + test_filesize(0x1000);
                    + test_filesize(0x10000);
                    + test_filesize(0x100000);
                    + test_filesize(0x1000000);

                    Where that largest testcase is 16 MiB.

                    I'm asking because i want to know if this syscall can be used to, for example, load icons and config files. Now config files are likely sub megabyte ones. But icons can be multiple megabytes (think about ico files that contain multiple images). But besides that, think about reading thumbnails for display in, say, dolphin or gwenview etc...

                    Now i'm not betting on this to be "substantially faster" then 3 syscalls (open/read/close) as the syscalls themselves are probably just a tiny percentage of actually handling the file data to do something with it (say jpeg/png/svg/webp decoder). But still, if you expand this over large folders it might become a notable difference.

                    Lastly, and this one is specifically for phoronix. Greg is, in the next version of this patch, going to post benchmarks too: https://lore.kernel.org/lkml/[email protected]/
                    There is no size limit, the mentioning of "small and medium" is because for smaller files the overhead of 3 syscalls is way larger than the actual time it takes to read the contents of the file and that is where this new syscall gives a performance boost, if it's convenience you are after then it's a boost regardless of file size.

                    Comment


                    • #20
                      Originally posted by F.Ultra View Post

                      There is no size limit, the mentioning of "small and medium" is because for smaller files the overhead of 3 syscalls is way larger than the actual time it takes to read the contents of the file and that is where this new syscall gives a performance boost, if it's convenience you are after then it's a boost regardless of file size.
                      Also there's the thing that you probably don't want to use this for gigabyte-sized files anyway as you might want to handle those in streamed way rather than reading everything into memory.

                      Comment

                      Working...
                      X