Announcement

Collapse
No announcement yet.

Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

    Phoronix: Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

    As part of David Howells of Red Hat long-term work on improving the caching code used by network file-systems, he today posted a big patch series rewriting the fscache and cachefiles code as the latest significant step on that adventure...

    https://www.phoronix.com/scan.php?pa...ache-cachefile

  • #2
    Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.

    Comment


    • #3
      I never thought there was a cache for networked file-systems...

      Comment


      • #4
        Simpler, but performance? benchmark?
        Developer of Ultracopier/Supercopier and of the game CatchChallenger

        Comment


        • #5
          Can it be updated to allow zfs integration. Its ridiculous that zfs occupies "in-use" memory.

          Comment


          • #6
            Originally posted by MadeUpName View Post
            Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.
            Its a very well known problem to Gnu/Linux..
            The page cache is not released, after being used, it can stay there forever, and your machine can start swapping like crazy, and no release of page cache..
            Or you can have a drop_cache cronjob that does it everyday, but even with that..if your system have to open a lot of files per interval of time.. you have no luck, in "less than nothing" the page cache is already full again, and the system starts to swap..
            tuxd3v
            Senior Member
            Last edited by tuxd3v; 29 November 2021, 07:23 PM.

            Comment


            • #7
              Thanks to David Howells for his work. If FS-Cache is for network file systems, the rewrite will benefit mostly servers? But rewriting CacheFiles might also benefit desktops? I'm genuinely curious.

              Comment


              • #8
                MadeUpName
                Senior Member
                MadeUpName and
                tuxd3v
                Senior Member
                tuxd3v , will these rewrites address the problems that you mention?

                Comment


                • #9
                  Originally posted by Eumaios View Post
                  MadeUpName
                  Senior Member
                  MadeUpName and
                  tuxd3v
                  Senior Member
                  tuxd3v , will these rewrites address the problems that you mention?
                  That is the real question, something that I am curious to test..

                  The Servers were I have seen this problems have NFS file systems, and local file systems, so I don't really know exactly where the problem reside, is related to network file systems, or local file systems..
                  However, I have also seen this problem in servers without NFS, I don't really know if this solves it..
                  In Kernel 2.4 I believe existed there a knob so that you could set the maximum amount of cache, that was very nice, because you could split your Ram into a big chunk for the application and let the rest for pagecache..
                  In kernel 2.6 that was retired..
                  This problem happen a lot when you need to read a lot of files, and they become cached, and then no way to retrieve that memory back( Unless explicitly.. ).. a lot of machines start swapping pages then..
                  The thing that I have done, and others also have done is a cronjob, set to drop the caches..

                  Supposedly this cache would have to return to the pool of free memory, after a while, but it doesn't or if it returns returns very few..

                  Comment


                  • #10
                    Originally posted by Eumaios View Post
                    MadeUpName
                    Senior Member
                    MadeUpName and
                    tuxd3v
                    Senior Member
                    tuxd3v , will these rewrites address the problems that you mention?
                    Perhaps indirectly. One of the things mentioned is moving the code into the kernel. Now normally I am loath to keep moving every thing into the kernel but I was actually surprised the code wasn't there already. One possibility is that with the kernel in charge the next time the kernel is feeling memory pressure, instead of dropping a kill -9 on the database it may ask "Hey I wonder if there is any of that dentry bloat I can let go of?".

                    Comment

                    Working...
                    X