Btrfs Zstd Compression Benchmarks On Linux 4.14

[victor.yarema]

do they support lz4? That would seem to be a good option as it shouldn't add any appreciable time either direction.

Taking into account the fact that fastest ZSTD can compress 470 MB/s while LZ4 can do 750 MB/s it seems to me that the difference of real life scenarios will be not noticeable. And at the same time ZSTD will provide better compression ratio. I guess LZ4 is good for things that are as fast as RAM. It is default for ZRAM. But for storage ZSTD wins for me.

[victor.yarema]

SSD in box that has SATA to USB controller, connected to laptop using USB2.0.
Hashing file using `pv Main.vdi | md5sum`.
"NTFS": 11,3GiB 0:04:44 [40,7MiB/s]
"BTRFS with compress-force=zstd:15": 11,3GiB 0:01:13 [ 158MiB/s]

[S.Pam]

phoronix can you test with compress-force=zstd:1 ? Zstd has much faster internal heuristics for skipping non-compressible data.

[Hi-Angel]

The system is a headless server, sorry, I can't help with these questions. The disks I have btrfs on are just big HDDs with videos and roms and stuff on that I access through Samba or NFS. I don't even remember whether the OS disk is formatted with btrfs.

zstd compression gave new life to BTRFS on my systems. Barely yesterday I installed Archlinux to BTRFS with enabled zstd compression to a laptop at work. Its HDD have the same speed 5400rpm like the one of mine own. But apps starting up noticeably faster, like, 2 or 2.5 times faster compared to my laptop at home with ext4. So, you know, I think I gonna use BTRFS with zstd for all new systems

[kaprikawn]

You're not really going to gain anything, perhaps even slow things down if your content is already highly compressed. If you store your ROMs uncompressed that can work, but then deduplication is an even bigger win if you have many similar ROMs.

I'm not overly concerned over the probably negligible difference in the access speed of accessing a ROM, most of which are probably only a few meg. Even if it's a disk image from PS1 or later, I doubt it'd make much difference. I don't know how much room I've saved on the video files, but my perception is that I've fitted a lot more onto btrfs than I used to get onto an EXTx formatted partition.

[Hi-Angel]

You're not really going to gain anything, perhaps even slow things down if your content is already highly compressed. If you store your ROMs uncompressed that can work, but then deduplication is an even bigger win if you have many similar ROMs.

BTRFS disables compression for files where compression gives no benefits. As of now it determined like this: BTRFS tries to compress first n bytes (don't remember the n), and if it didn't work, marks the file as useless to compress. It sometimes gives false positives, so prone to change in the future — but at least it does sort out cases like you mentioned.

[s_j_newbury]

The system is a headless server, sorry, I can't help with these questions. The disks I have btrfs on are just big HDDs with videos and roms and stuff on that I access through Samba or NFS. I don't even remember whether the OS disk is formatted with btrfs.

You're not really going to gain anything, perhaps even slow things down if your content is already highly compressed. If you store your ROMs uncompressed that can work, but then deduplication is an even bigger win if you have many similar ROMs.

[kaprikawn]

Do you notice any improvement in startup times of the system or maybe apps as opposed to no compression at all?

The system is a headless server, sorry, I can't help with these questions. The disks I have btrfs on are just big HDDs with videos and roms and stuff on that I access through Samba or NFS. I don't even remember whether the OS disk is formatted with btrfs.

[Hi-Angel]

I've got a server which primarily is a file server, CPU usage and the like aren't my major concerns, compression rates are. When the amount of data you're using is measured in terabytes you can fit a lot more data on if the compression technique is more aggressive.

I started out with lzo because I didn't know any better when I first used btrfs, I just saw it on the Arch wiki or an online tutorial, tried it and it worked (and I got noticeably more data on my disks). I've since enabled zlib on the latest disk I added more recently. I'm very happy with the results.

Do you notice any improvement in startup times of the system or maybe apps as opposed to no compression at all?

[Hi-Angel]

There was a patch enabling lz4 floating around for a while. I gave it a go and found it worked well, but lz4hc support was buggy and would fail on decompression at times. I believe, because of the unresolved lz4hc issue which presumably could have also affected standard compression although it never showed up, the code wasn't accepted upstream.

From the btrfs wiki:

The LZ4 algorithm was considered but has not brought significant gains.