Major Rewrite Of Linux's FS-Cache / CacheFiles So It's Smaller & Simpler

[cl333r]

Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.

Not an expert on any on this but I was wondering if there should be a totally new POSIX 2.0 (like we did with Vulkan vs GL, or Wayland vs X11), it shows its age by dealing with its old naming conventions from the 1980's and little issues like that. I for one don't like that when you list files in any folder the system also lists "." and "..", so 99.9% of code has to test each file name against these names:

for (...) {
if (d->d_name != "." && d->d_name != "..")
// go on
}

[tuxd3v]

Thanks to David Howells for his work. If FS-Cache is for network file systems, the rewrite will benefit mostly servers? But rewriting CacheFiles might also benefit desktops? I'm genuinely curious.

CacheFiles files should benefit for cache from local filesystems..so it should theoretically also benefit desktop's,
The amount of the benefit its something that we don't know, it will depend from case to case, considering each use case,
But this, if CacheFiles is what I am thinking about, I am not sure..

Maybe someone more educated on the matter can elaborate more about the subject..

[MadeUpName]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

Perhaps indirectly. One of the things mentioned is moving the code into the kernel. Now normally I am loath to keep moving every thing into the kernel but I was actually surprised the code wasn't there already. One possibility is that with the kernel in charge the next time the kernel is feeling memory pressure, instead of dropping a kill -9 on the database it may ask "Hey I wonder if there is any of that dentry bloat I can let go of?".

[tuxd3v]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

That is the real question, something that I am curious to test..

The Servers were I have seen this problems have NFS file systems, and local file systems, so I don't really know exactly where the problem reside, is related to network file systems, or local file systems..
However, I have also seen this problem in servers without NFS, I don't really know if this solves it..
In Kernel 2.4 I believe existed there a knob so that you could set the maximum amount of cache, that was very nice, because you could split your Ram into a big chunk for the application and let the rest for pagecache..
In kernel 2.6 that was retired..
This problem happen a lot when you need to read a lot of files, and they become cached, and then no way to retrieve that memory back( Unless explicitly.. ).. a lot of machines start swapping pages then..
The thing that I have done, and others also have done is a cronjob, set to drop the caches..

Supposedly this cache would have to return to the pool of free memory, after a while, but it doesn't or if it returns returns very few..

[Eumaios]

MadeUpName and tuxd3v , will these rewrites address the problems that you mention?

[Eumaios]

Thanks to David Howells for his work. If FS-Cache is for network file systems, the rewrite will benefit mostly servers? But rewriting CacheFiles might also benefit desktops? I'm genuinely curious.

[tuxd3v]

Hopefully this leads to the fs dentry cache not eating so much RAM. We had a file system where every time we did an rsync it would use 32GB of RAM and not give it up until we did a manual flush. For long term file storage systems where a file may only be really accessed once a year it need to learn to let go.

Its a very well known problem to Gnu/Linux..
The page cache is not released, after being used, it can stay there forever, and your machine can start swapping like crazy, and no release of page cache..
Or you can have a drop_cache cronjob that does it everyday, but even with that..if your system have to open a lot of files per interval of time.. you have no luck, in "less than nothing" the page cache is already full again, and the system starts to swap..

[down1]

Can it be updated to allow zfs integration. Its ridiculous that zfs occupies "in-use" memory.

[alpha_one_x86]

Simpler, but performance? benchmark?

[tildearrow]

I never thought there was a cache for networked file-systems...