Announcement

Collapse
No announcement yet.

New readfile() System Call Under Review For Reading Small~Medium Files Faster

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    markg85 whoever said I was using a Pi? It was an i.MX 5 SoM. And yes, fast switching was kind of stupid, but our client wanted it so what was I to do? Also, on a single core 500 MHz Cortex-A7 I'll take all the savings I can get.

    Comment


    • #12
      Originally posted by jaskij View Post
      markg85 whoever said I was using a Pi? It was an i.MX 5 SoM. And yes, fast switching was kind of stupid, but our client wanted it so what was I to do? Also, on a single core 500 MHz Cortex-A7 I'll take all the savings I can get.
      Oops, i messed up there. I read @milkylainen post first (which mentions the raspberry pi and was a reply to yours) and blindly assumed that you were using a raspberry pi. I did read your post but just assumed the wrong thing. Sorry for that.

      I stand by my suggestion that you should probably need to go for an RTOS system. Also, who knows better? You, the developer, or your client? I'm going to assume (again) that you know better and should convince your client of the better approach.

      Comment


      • #13
        markg85 nowadays I would. But I was a noob who got thrown into a middle of a project. The hardware was done, the client had stupid requirements and I was a noob trying to make the best out of it.

        These days I'd probably tell them to just slap a hardware PWM or an MCU in there. But if you're doing small runs anything software-defined is usually not worth the added R&D costs if you can pay extra 5$/device in hardware.

        Also, I stand by my position that for cheap stuff with a 500 MHz Cortex-A7 I'll take all the savings I can get.

        P.S.
        Sorry for not quoting, Phoronix forums are iffy on mobile.

        Comment


        • #14
          Originally posted by atomsymbol
          Is there a real-world scenario in which open/read/close of sysfs and procfs files is a bottleneck?
          Yes, there is.

          Comment


          • #15
            Originally posted by atomsymbol
            Without the ability to read multiple small files using a single system call, it is impossible to increase IOPS (unless using multiple threads).
            IOPS is Io Operations Per Second. You can increase them either by overlapping more operations or decreasing the time between successive operations. This addresses only the latter, though still improves performance (which, again, is their reason for doing it)

            If you also want the former, then you can simply use it with io_uring to queue up multiple readfile() operations.

            So, you can have your cake and eat it!

            Comment


            • #16
              Originally posted by markg85 View Post
              Can anyone tell me the maximum filesize for this syscall?
              Yes, i did read the code, and the new manpage. Nothing seems to indicate what qualifies as "small". The post also mentions "small and medium sizes".

              It even has testcases:

              + test_filesize(0x10);
              + test_filesize(0x100);
              + test_filesize(0x1000);
              + test_filesize(0x10000);
              + test_filesize(0x100000);
              + test_filesize(0x1000000);

              Where that largest testcase is 16 MiB.

              I'm asking because i want to know if this syscall can be used to, for example, load icons and config files. Now config files are likely sub megabyte ones. But icons can be multiple megabytes (think about ico files that contain multiple images). But besides that, think about reading thumbnails for display in, say, dolphin or gwenview etc...

              Now i'm not betting on this to be "substantially faster" then 3 syscalls (open/read/close) as the syscalls themselves are probably just a tiny percentage of actually handling the file data to do something with it (say jpeg/png/svg/webp decoder). But still, if you expand this over large folders it might become a notable difference.

              Lastly, and this one is specifically for phoronix. Greg is, in the next version of this patch, going to post benchmarks too: https://lore.kernel.org/lkml/[email protected]/
              There is no size limit, the mentioning of "small and medium" is because for smaller files the overhead of 3 syscalls is way larger than the actual time it takes to read the contents of the file and that is where this new syscall gives a performance boost, if it's convenience you are after then it's a boost regardless of file size.

              Comment


              • #17
                Originally posted by F.Ultra View Post

                There is no size limit, the mentioning of "small and medium" is because for smaller files the overhead of 3 syscalls is way larger than the actual time it takes to read the contents of the file and that is where this new syscall gives a performance boost, if it's convenience you are after then it's a boost regardless of file size.
                Also there's the thing that you probably don't want to use this for gigabyte-sized files anyway as you might want to handle those in streamed way rather than reading everything into memory.

                Comment


                • #18
                  Originally posted by F.Ultra View Post

                  There is no size limit, the mentioning of "small and medium" is because for smaller files the overhead of 3 syscalls is way larger than the actual time it takes to read the contents of the file and that is where this new syscall gives a performance boost, if it's convenience you are after then it's a boost regardless of file size.
                  Ohh, that changes things! Thank you for that reply!
                  That means that any file you'd previously read into memory could now use this syscall. Or rather, should! As it only give you a net win.

                  Now i kinda hope that toolkits (say Qt/GTK) are going to provide support for this too!

                  Those places that already did open/read themselves with smarter memory usage (like streaming) should just stay as is i think?

                  Comment


                  • #19
                    Originally posted by atomsymbol
                    Just a note: Reducing 3 syscalls into 1 most likely won't increase uncached IOPS: HDD has about 100 random IOPS, SSD is about 10000 random IOPS (except for 3D XPoint), a getpid() syscall is about 100 nanoseconds and 1/100ns = 10 million no-operation "IOPS". 10e6 is 1000 times smaller than 10e3.
                    Good point.

                    On what CPU were those measurements taken? Did you have any mitigations disabled? Not that it would change the overall impact on IOPS (except perhaps with Optane), but it'd be good to know.

                    Comment


                    • #20
                      Originally posted by atomsymbol

                      Ryzen 3000 desktop. Mitigations disabled.
                      then your results are worth nothing, sorry
                      While i agree with disabling mitigations. Everyone should do that! The security paranoia went too far on that front. The super majority will just update their system and don't bother adding boot flags to disable them. Distributions also likely won't add them..

                      You're also comparing roughly the slowest medium for these days (spinning drives). Lots of people nowadays have an NVMe as their daily driver. That will significantly change your results. Still a syscall is likely going to be a very minor part of the whole operation that was executed. So for one call that won't give any notable performance difference i suppose. But things might change if you multiply it by tens of thousands.. Remember, a simple action like showing a thumbnail probably executes a handful of syscalls now (stat it, read, open, close, extended attributes, ....).

                      Comment

                      Working...
                      X