Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

[phoronix]

Phoronix: Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

As part of David Howells of Red Hat long-term work on improving the caching code used by network file-systems, he today posted a big patch series rewriting the fscache and cachefiles code as the latest significant step on that adventure...

https://www.phoronix.com/scan.php?pa...ache-cachefile

[MadeUpName]

Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.

[tildearrow]

I never thought there was a cache for networked file-systems...

[alpha_one_x86]

Simpler, but performance? benchmark?

[down1]

Can it be updated to allow zfs integration. Its ridiculous that zfs occupies "in-use" memory.

[tuxd3v]

Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.

Its a very well known problem to Gnu/Linux..
The page cache is not released, after being used, it can stay there forever, and your machine can start swapping like crazy, and no release of page cache..
Or you can have a drop_cache cronjob that does it everyday, but even with that..if your system have to open a lot of files per interval of time.. you have no luck, in "less than nothing" the page cache is already full again, and the system starts to swap..

[Eumaios]

Thanks to David Howells for his work. If FS-Cache is for network file systems, the rewrite will benefit mostly servers? But rewriting CacheFiles might also benefit desktops? I'm genuinely curious.

[Eumaios]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

[tuxd3v]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

That is the real question, something that I am curious to test..

The Servers were I have seen this problems have NFS file systems, and local file systems, so I don't really know exactly where the problem reside, is related to network file systems, or local file systems..
However, I have also seen this problem in servers without NFS, I don't really know if this solves it..
In Kernel 2.4 I believe existed there a knob so that you could set the maximum amount of cache, that was very nice, because you could split your Ram into a big chunk for the application and let the rest for pagecache..
In kernel 2.6 that was retired..
This problem happen a lot when you need to read a lot of files, and they become cached, and then no way to retrieve that memory back( Unless explicitly.. ).. a lot of machines start swapping pages then..
The thing that I have done, and others also have done is a cronjob, set to drop the caches..

Supposedly this cache would have to return to the pool of free memory, after a while, but it doesn't or if it returns returns very few..

[MadeUpName]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

Perhaps indirectly. One of the things mentioned is moving the code into the kernel. Now normally I am loath to keep moving every thing into the kernel but I was actually surprised the code wasn't there already. One possibility is that with the kernel in charge the next time the kernel is feeling memory pressure, instead of dropping a kill -9 on the database it may ask "Hey I wonder if there is any of that dentry bloat I can let go of?".