OpenZFS Could Soon See Much Better Deduplication Support

[linuxgeex]

My zfs systems are very peculiar, because they run on Optane drives and a 512/4K recordsize/sectorsize: http://www.linuxsystems.it/2018/05/o...t4-benchmarks/

With those settings ram usage with deduplication would be WAAAAAAY higher and even compression is less effective.

General case 32k record size is good because it gives the codec some context to get a good ratio.

For dedupe, larger record size = lower RAM usage, until the record size approaches the median file size.

For massive multi-hosted VPN, 4k record size is the best. If you're expanding 32k and using 4k of it, that's pretty hard on both the pagecache and CPU cache. Throwing away a bit of storage capacity so that you don't tank when things are getting difficult, is a good tradeoff.

[darkbasic]

I am always getting mixed answers to how much ram openzfs needs (from atleast 8gb to not much unless you use specific features) . Have you it on some 1-2 GB systems?

My zfs systems are very peculiar, because they run on Optane drives and a 512/4K recordsize/sectorsize: http://www.linuxsystems.it/2018/05/o...t4-benchmarks/

With those settings ram usage with deduplication would be WAAAAAAY higher and even compression is less effective.

[timofonic]

I just want persistent cache.

What's that? And why do you need it?

[dfyt]

I just want persistent cache.

[starshipeleven]

I am always getting mixed answers to how much ram openzfs needs (from atleast 8gb to not much unless you use specific features) . Have you it on some 1-2 GB systems?

It will work for home-sized arrays (a few TB)
https://serverfault.com/questions/74...ool-block-size
https://github.com/zfsonlinux/zfs/issues/5417
as long as you don't use dedup and L2ARC.

Having more RAM is better because it's used as cache, but you don't need that much RAM to just use ZFS.

Apart from that, there is a lot of legends and hearsay about heavy RAM usage/requirements coming from FreeNAS (FreeBSD) people that does not apply to OpenZFS (and soon will not apply to FreeNAS/BSD anymore too as they are migrating to OpenZFS codebase for their ZFS)
Pool block size in OpenZFS is huge if compared to Solaris and FreeBSD, and they have reduced massively the RAM requirement for L2ARC.

See my post and the answers here https://www.phoronix.com/forums/foru...15#post1112115

[discordian]

I already use zfs in lots of machines, but If this really works well without requiring a TB of ram then I'll probably switch all my installations from btrfs to zfs.

I am always getting mixed answers to how much ram openzfs needs (from atleast 8gb to not much unless you use specific features) . Have you it on some 1-2 GB systems?

[starshipeleven]

Damn me and my timing, I just bought a Fujitsu rack server that can mount up to 768 GB of ECC RAM.

[darkbasic]

I already use zfs in lots of machines, but If this really works well without requiring a TB of ram then I'll probably switch all my installations from btrfs to zfs.

[k1e0x]

That's great.

Dedup would be really great if it worked and it was cheep. (like compression).

[gorgone]

My proxmox will love this. I have alot(10+) of debian containers ... this will massively reduce the size.