Announcement

Collapse
No announcement yet.

New readfile() System Call Under Review For Reading Small~Medium Files Faster

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • New readfile() System Call Under Review For Reading Small~Medium Files Faster

    Phoronix: New readfile() System Call Under Review For Reading Small~Medium Files Faster

    Back in May we reported on work being done for a readfile() system call to read small files more efficiently. Greg Kroah-Hartman has now volleyed those patches for review on the kernel mailing list for this improvement for reading small to medium file sizes on Linux systems...

    http://www.phoronix.com/scan.php?pag...ys-Call-Review

  • #2
    Originally posted by phoronix View Post
    Phoronix: New readfile() System Call Under Review For Reading Small~Medium Files Faster

    Back in May we reported on work being done for a readfile() system call to read small files more efficiently. Greg Kroah-Hartman has now volleyed those patches for review on the kernel mailing list for this improvement for reading small to medium file sizes on Linux systems...

    http://www.phoronix.com/scan.php?pag...ys-Call-Review
    At first, I thought that the proposed system call is capable of reading multiple small files using a single system call - which would help increase HDD/SSD queue utilization and increase IOPS (I/O operations per second) - but that isn't the case and the proposed system call can read just a single file. Without the ability to read multiple small files using a single system call, it is impossible to increase IOPS (unless using multiple threads).

    https://en.wikipedia.org/wiki/Native_Command_Queuing

    Comment


    • #3
      For everybody, who is also wondering why you'd need to pass a file descriptor : it is the file descriptor of the directory this file lives in. You've to 'open' the directory first. Source : patches

      Comment


      • #4
        Originally posted by atomsymbol View Post

        At first, I thought that the proposed system call is capable of reading multiple small files using a single system call - which would help increase HDD/SSD queue utilization and increase IOPS (I/O operations per second) - but that isn't the case and the proposed system call can read just a single file. Without the ability to read multiple small files using a single system call, it is impossible to increase IOPS (unless using multiple threads).

        https://en.wikipedia.org/wiki/Native_Command_Queuing
        It is not too late, comment those patches on the mailing list

        Comment


        • #5
          Originally posted by atomsymbol View Post

          At first, I thought that the proposed system call is capable of reading multiple small files using a single system call - which would help increase HDD/SSD queue utilization and increase IOPS (I/O operations per second) - but that isn't the case and the proposed system call can read just a single file. Without the ability to read multiple small files using a single system call, it is impossible to increase IOPS (unless using multiple threads).

          https://en.wikipedia.org/wiki/Native_Command_Queuing
          There are no queues and iops in sysfs and procfs This syscall is not about better hardware utilization, it's about doing fewer syscalls.

          If you are at the point of being bothered by queue utilization, then you should already be using async I/O (cf. io_uring).

          Comment


          • #6
            Originally posted by oleid View Post
            For everybody, who is also wondering why you'd need to pass a file descriptor : it is the file descriptor of the directory this file lives in. You've to 'open' the directory first. Source : patches
            I can think of two reasons to do it that way off the top of my head:
            1. Linux has no path length limit. Some legacy POSIX APIs are limited to 4096 characters, but filesystems like ext4 have no path length limit and you can work around the API limits using relative paths.
            2. It's getting more popular to compile code to ABIs where all paths must be relative to file descriptors passed in by the sandbox host. (i.e. The binary does not have access to any syscalls that accept absolute paths)

            Comment


            • #7
              Originally posted by intelfx View Post
              There are no queues and iops in sysfs and procfs This syscall is not about better hardware utilization, it's about doing fewer syscalls.
              Is there a real-world scenario in which open/read/close of sysfs and procfs files is a bottleneck?

              Originally posted by intelfx View Post
              If you are at the point of being bothered by queue utilization, then you should already be using async I/O (cf. io_uring).
              I agree.

              Comment


              • #8
                Originally posted by oleid View Post
                It is not too late, comment those patches on the mailing list
                Done that. I suppose the kernel mailing list is getting proposals for new syscalls like readfile() multiple times per year and is rejecting them most of the time ....

                Comment


                • #9
                  Originally posted by atomsymbol View Post
                  Is there a real-world scenario in which open/read/close of sysfs and procfs files is a bottleneck?
                  Not sure — but I didn't make this up, it's just the rationale from the lkml thread:

                  <...>
                  This is especially helpful for tools that poke around in procfs or sysfs, making a little bit of a less system load than before, especially as syscall overheads go up over time due to various CPU bugs being addressed.
                  <...>

                  Comment


                  • #10
                    atomsymbol I've run precisely into this sysfs bottleneck when doing fast switching of GPIOs in an industrial PC. I ended up keeping those fds open and doing seek()/write() over open()/write()/close() - it cut down my CPU utilization by 90%.

                    ​​​​

                    Comment

                    Working...
                    X