Announcement

Collapse
No announcement yet.

New readfile() System Call Under Review For Reading Small~Medium Files Faster

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • coder
    replied
    Originally posted by atomsymbol View Post
    It has less than 0.5% CPU utilization on my machine with 183 tasks, 218 kernel threads and 1538 total threads. htop: 1.3%.
    Still, it must be kept very low, so that you don't impact the performance of the machine you're trying to monitor. And my primary concern would be for embedded use cases, where I think I've seen it use like 10% or more.

    Leave a comment:


  • atomsymbol
    replied
    Originally posted by markg85 View Post
    We OBVIOUSLY mean loading from disk.
    I never explicitly stated (you can check the record if you want) that it is about pushing 1GB/s of perfectly-random data through a 600MB/s SATA physical connection. The assumption that I was talking about transmitting 1GB/s through a 600MB/s connection was an extra invention/assumption of your own. If you had not pushed the discussion towards the obviously impossible, the discussion would have been much more fruitful.

    Leave a comment:


  • atomsymbol
    replied
    Originally posted by rene View Post
    IMHO quite waste of kernel code and developer time to implement such a highly specific system call for not a too high performance nor mission critical code. Instead I would rather like to see generically vectored / bundled systemically, like my already previously proposed syscallv to submit any combination of open, read, write, close, and any other arbitrary groups of system calls that make way more sense for high performance applications from databases, to fileservers and the like: https://www.youtube.com/watch?v=84Uyh3KwY-w
    syscallv would be very nice. The same idea crossed my mind, thought I didn't post it.

    Leave a comment:


  • rene
    replied
    IMHO quite waste of kernel code and developer time to implement such a highly specific system call for not a too high performance nor mission critical code. Instead I would rather like to see generically vectored / bundled systemically, like my already previously proposed syscallv to submit any combination of open, read, write, close, and any other arbitrary groups of system calls that make way more sense for high performance applications from databases, to fileservers and the like: https://www.youtube.com/watch?v=84Uyh3KwY-w

    Leave a comment:


  • markg85
    replied
    Originally posted by atomsymbol View Post

    In my opinion, you are missing quite a lot by not ever reading the NaCl paper. It's just a single paper, not a hundred papers.



    Well, how about this: If a file is cached in memory and its size is 1 byte then I am claiming it is possible to achieve a speedup of 100x.
    Seriously.. That's how you plan on winning this argument.
    By cheating...

    We OBVIOUSLY mean loading from disk.
    End of discussion. Your arguments are without proof. I'm done responding to this.

    Leave a comment:


  • atomsymbol
    replied
    Originally posted by markg85 View Post

    You claim something has a 10x improvement.
    You expect me to read all there is about NaCl without giving me a single hint as to where i'm even supposed to look.

    No deal!
    In my opinion, you are missing quite a lot by not ever reading the NaCl paper. It's just a single paper, not a hundred papers.

    Originally posted by markg85 View Post
    I am dismissing NaCl as i'm not going to read up hundreds of articles and long videos to "verify your claim". I'm not here for that.
    Till you prove your point with numbers and articles to back it up, i'm going to downright call your claims total bullshit.

    It would be theoretically impossible to get a 10x improvement on the low level file reading where the hardware remains as is and just one part of the software stack changes. It "would" be possible to make that claim in end user libraries where there are loads of indirections in between all making it a bit slower. But at the low level: bullshit. Just remember, again, that even io_uring isn't improving performance much. Only by a small margin, say 30 percent max. Very far from 10x, more like 0.3x. The benefit, yet again, is in the async nature.
    Well, how about this: If a file is cached in memory and its size is 1 byte then I am claiming it is possible to achieve a speedup of 100x.

    Leave a comment:


  • markg85
    replied
    Originally posted by atomsymbol View Post

    NaCl is part of a research area. A person can choose to read articles from that research area or choose not to read them. Dismissing a whole research area based on literally just a couple of sentences from a web search engine isn't a good strategy.

    If you are familiar with the research area to which NaCl belongs then please reply with an argument why you believe that the core idea behind NaCl cannot be used to achieve the speedup I am claiming can be achieved.


    You claim something has a 10x improvement.
    You expect me to read all there is about NaCl without giving me a single hint as to where i'm even supposed to look.

    No deal!

    I am dismissing NaCl as i'm not going to read up hundreds of articles and long videos to "verify your claim". I'm not here for that.
    Till you prove your point with numbers and articles to back it up, i'm going to downright call your claims total bullshit.

    It would be theoretically impossible to get a 10x improvement on the low level file reading where the hardware remains as is and just one part of the software stack changes. It "would" be possible to make that claim in end user libraries where there are loads of indirections in between all making it a bit slower. But at the low level: bullshit. Just remember, again, that even io_uring isn't improving performance much. Only by a small margin, say 30 percent max. Very far from 10x, more like 0.3x. The benefit, yet again, is in the async nature.

    Leave a comment:


  • atomsymbol
    replied
    Originally posted by markg85 View Post
    Enlighten me again and this time with links and research to back your claims up.
    I call your claim bullshit unless proven wrong.

    Your credibility with me is declining super fast now.
    If i search for NaCl all i find is "Google Native Client" which has absolutely nothing to do with this conversation.
    NaCl is part of a research area. A person can choose to read articles from that research area or choose not to read them. Dismissing a whole research area based on literally just a couple of sentences from a web search engine isn't a good strategy.

    If you are familiar with the research area to which NaCl belongs then please reply with an argument why you believe that the core idea behind NaCl cannot be used to achieve the speedup I am claiming can be achieved.

    Originally posted by markg85 View Post
    So bring proof or just don't post. As this is getting tiresome.

    Note, there actually is one system that could get up to 10x the performance. That's the playstation 5, but that again has nothing to do with this. I was asking for linux and now. Not hypothetical or other platforms.

    Leave a comment:


  • ssokolow
    replied
    Originally posted by pal666 View Post
    it's api, not abi. there are two reasons: it's faster and it avoids toctou
    I meant what I said. I was referring to how, if you were to hack together some machine code by hand, the execution environment still wouldn't allow you to make calls outside the set described by the API and, in idealized form, would lack the "vocabulary" to describe such calls.

    Leave a comment:


  • markg85
    replied
    Originally posted by atomsymbol View Post

    NaCl.
    Enlighten me again and this time with links and research to back your claims up.
    I call your claim bullshit unless proven wrong.

    Your credibility with me is declining super fast now.
    If i search for NaCl all i find is "Google Native Client" which has absolutely nothing to do with this conversation.

    So bring proof or just don't post. As this is getting tiresome.

    Note, there actually is one system that could get up to 10x the performance. That's the playstation 5, but that again has nothing to do with this. I was asking for linux and now. Not hypothetical or other platforms.

    Leave a comment:

Working...
X