Facebook Looking To Add Zstd Support To The Linux Kernel, Btrfs

[timofonic]

Of course when someone comes up with a great new algorithms, fucking patents ruin everything. So it will be possible to use this in 2041?

If Facebook want Zstd in the vanilla Linux kernel, they should ask pardon and remove that crap in the license....

[timofonic]

I've been off facebook for years, for my own reasons. But these days, they are into heavy censorship and thought control. I cannot find a more manipulative company. Apple looks like saints in comparison.

Maybe they should read Google's simple three words of wisdom: "Do no evil".

Remember how they failed their HTML5 absolute commitment?

Google has been quite evil since some time ago, even before Alphabet (but this movement made them more evil, of course). They are doing evil things in Android, making more and more subsystems closed source by the Google Play Services and others. And there's more to say, but I prefer to not say it...

[name99]

Any word on how this might compare to xz?

It's trying to solve a completely different problem. xz is trying for maximum compression taking LOTS of time --- appropriate for file archiving. zstd is trying for good enough compression in real time (with a fraction of a CPU) for things like on-the-fly HTTP compression (or memory page compression? maybe that is so time sensitive it always uses LZ4?)
A better comparison is to Apple's LZFSE which was released a year earlier (2015 rather than 2016), also has an open-source version, and has much the same overall characteristics (performance, compression ratio, energy use and memory use) which isn't surprising since they're both using much the same ideas.

https://lyncd.com/2015/09/lossless-c...on-innovation/

[profoundWHALE]

you should prefer no compression at all then, it is infinitely faster than lz4. smart people though care about total time of read + decompress and, depending on drive and cpu, lzma could be fastest

I don't understand. Have you ever used ZFS?

[terrelln]

One big problem with zstd, that I just can't see the Linux kernel team being happy with, is that over-reaching PATENT agreement. It's not really accurate to call zstd BSD licensed. It's BSD + a whole big "You can't claim patent infringement against us, ever"

The zstd library in the kernel patch is dual licensed with GPLv2 https://github.com/facebook/zstd/blo...e/linux/zstd.h.

wait, does that mean facebook isn't going to start contributing to zfs with its wonderful lz4 compression?

The lz4 library is licensed under the vanilla BSD 2-clause license https://github.com/lz4/lz4/blob/dev/LICENSE.

[starshipeleven]

Google has been quite evil since some time ago, even before Alphabet (but this movement made them more evil, of course). They are doing evil things in Android, making more and more subsystems closed source by the Google Play Services and others. And there's more to say, but I prefer to not say it...

Not seeing the evilness. By rolling many functions as Google Play Services instead than relying on OS's stuff, is that they can keep stuff patched and safe. Same for the system's integrated browser component (that now gets updates just like any other app, and it's very fucking good to have it like that).

Really, the main Android issue isn't Google rolling its own closed stuff (easily replaceable by a shim, there is a guy that made a Play Services shim for example) in what is usually a completely closed source firmware based off AOSP, it's OEMs/carriers/hardware manufacturers that ship closed source stuff in the kernel or refuse to give sources when they should, so perfectly fine hardware can't be freedomized with a custom firmware and kept up-to-date on the software side.

[GI_Jack]

Next big real question: What is the patent situtation with Zstd? Is it patent uncumbered? Or is facebook trying to push something they own patents to.

Not the first time they've done that. The code itself is free, but the technology is covered by patents they can sue you over.

edit:

https://github.com/facebook/zstd/blob/dev/PATENTS

from earlier in the thread. Unless facebook is willing to drop patents, this can kindly stay away from the linux kernel, potentially entrapping any linux user..

[AndyChow]

I just tested it for myself. It's better than LZMA, but definitely not faster than zip. In fact, my tests show it's slower than LZMA, Still, interesting. I can't reproduce the benchmarks, by far.

[terrelln]

I just tested it for myself. It's better than LZMA, but definitely not faster than zip. In fact, my tests show it's slower than LZMA, Still, interesting. I can't reproduce the benchmarks, by far.

Zstd's highest compression levels (15-19) are expected to be around the speed of LZMA, with slightly worse compression. Its mid-levels (5-15) range zlib's compression speed, but with better compression ratio on most files. Its low levels (1-5) are expected to be faster than zlib (but not quite as fast as lz4), with zlib-esqe compression ratio. zstd level 3 has comparable compression to zlib level 6.

What zstd levels are you testing?
What system are you testing on?
Are you testing the kernel version or the zstd library/CLI?

[Windingroads]

I seriously hope nothing from face book ever gets in the kernel, these people are always trying to get new ways into our systems.