OpenZFS File-System Merges Support For Using Zstd Compression

[phoronix]

Phoronix: OpenZFS File-System Merges Support For Using Zstd Compression

Adding to the list of software projects embracing Zstd for its very efficient compression abilities, OpenZFS is now supporting the Facebook-developed Zstandard as its latest file-system compression support...

http://www.phoronix.com/scan.php?page=news_item&px=OpenZFS-Merges-Zstd

[Ornias1993]

Great to finally have it merged...
Special thanks to:
Allan Jude for the Original PoC and last month of bug fixes
Sebastian Gottschall for keeping it alive, fixing all of the bugs in the PoC, rewrithing the Allocation code, porting, testing and bugtracing
Michael Niewöhner for complete cleanup, Allocation code improvements and getting this PR in a mergable state

[peterdk]

Nice, looking forward to using this!

Now I am waiting for ZSTD support for TokuDB, but it seems that development on that DB has been almost stopped.

[uid313]

Nice to see (Open)ZFS evolve with new features and algorithms.

I wonder how the future file systems will look in ten or twenty years from now. In a world without SATA, SCSI, SAS and mechanical HDD.

[waxhead]

Nice to see (Open)ZFS evolve with new features and algorithms.

I wonder how the future file systems will look in ten or twenty years from now. In a world without SATA, SCSI, SAS and mechanical HDD.

Probably it will be a database (which a filesystem really is).

[curfew]

waxhead Yes, filesystems are already databases. Therefore your statement makes very little sense.

[waxhead]

waxhead Yes, filesystems are already databases. Therefore your statement makes very little sense.

So when block devices are a thing of the past and everything is a chunks of non-volatile memory does it make sense then?

[oiaohm]

So when block devices are a thing of the past and everything is a chunks of non-volatile memory does it make sense then?

https://zonedstorage.io/linux/fs/

As insane as it sounds XFS is being worked on behave correctly on zone based storage. The reality is all consume SSD are chunks of non-volatile storage that due to controller magic appear to be block device. Linux kernel has dm-zoned that is basically a cpu run equal to what has been in your SSD drive controller. This is really like the winmodem thing again.

File systems optimised for chunks of storage will perform better in future on zoned based devices. But file systems not optimised for chucks of storage will still work behind stuff like dm-zoned at a price of higher cpu usage.

https://nvdimm.wiki.kernel.org/

There is another bit of tech to remember about NVDIMM. NVDIMM go the other way to zoned based devices. NVDIMM lot of cases you end with ram storage equal to flash storage with a supercap to provide power to write to flash if/when power is lost. Yes these current have a virtual block device on top of them.

Between dm-zoned and NVDIMM virtual block device stuff block device is going to be around well after we no longer have block devices in hardware. Of course most likely not the best performing choice using a file system designed for only block device.

[k1e0x]

Nice to see (Open)ZFS evolve with new features and algorithms.

I wonder how the future file systems will look in ten or twenty years from now. In a world without SATA, SCSI, SAS and mechanical HDD.

https://zonedstorage.io/linux/fs/

As insane as it sounds XFS is being worked on behave correctly on zone based storage. The reality is all consume SSD are chunks of non-volatile storage that due to controller magic appear to be block device. Linux kernel has dm-zoned that is basically a cpu run equal to what has been in your SSD drive controller. This is really like the winmodem thing again.

File systems optimised for chunks of storage will perform better in future on zoned based devices. But file systems not optimised for chucks of storage will still work behind stuff like dm-zoned at a price of higher cpu usage.

https://nvdimm.wiki.kernel.org/

There is another bit of tech to remember about NVDIMM. NVDIMM go the other way to zoned based devices. NVDIMM lot of cases you end with ram storage equal to flash storage with a supercap to provide power to write to flash if/when power is lost. Yes these current have a virtual block device on top of them.

Between dm-zoned and NVDIMM virtual block device stuff block device is going to be around well after we no longer have block devices in hardware. Of course most likely not the best performing choice using a file system designed for only block device.

That sounds like Linux. To me it just seems like a waste of time to take a filesystem from the 80's and add this. If that is the case.. we can just do something new. The right tool for the right job.

[oiaohm]

That sounds like Linux. To me it just seems like a waste of time to take a filesystem from the 80's and add this. If that is the case.. we can just do something new. The right tool for the right job.

Its not the 1980s. XFS is 1993 when it first comes into existence. Object-Based storage idea starts being mucked around in 1995 while XFS core structures are still being made.

2004 Object-Based Storage Devices - 2 (OSD-2) feature for scsi. Interest one of the last feature in the IRIX release in 2006 was support of OSD-2 in IRIX XFS.

Yes the way you have to work on zoned file systems and OSD-2 systems is so the same that you don't in fact need new structures just use different drive commands.

https://en.wikipedia.org/wiki/XFS

Its really interesting what features XFS is meant to have. There is a large section of XFS that just has never been implemented in Linux. Yes a almost 30 year old file system and Linux kernel has not implemented it all yet.

Zoned is not a 100 percent new idea. If XFS truly from the 80s before object based storage work or was a file system not design with object based storage support making it support zoned would be a lot of work. XFS is most likely the oldest file-system that the features you need to support zoned is in fact built into the base file system structures there are a lot of younger file systems without the features.

k1e0x there are the fun things where its not add something new but implement something that was designed into the file system a long time ago that the Linux kernel implementation did not implement.

Also remember XFS was always designed to work hand and hand with the volume manager. Linux kernel only implemented the XFS but not its matching volume manager. XFS with XLV(its volume manager) is meant to have on the fly snapshots and copy on write and RAID like ZFS. Stupid as it sounds ZFS Zpool is really like XLV feature set.

XFS is not that primitive as it first appears it would help if more than half the features XFS/XLV was meant to have was in fact implemented in Linux.

k1e0x I guess you would not be thinking Hierarchical storage management is a XFS/XLV feature yes XFS/XLV is meant to have a L2ARC equal where it put the most recent accessed stuff on faster storage.

The reality the feature differences between ZFS vs XFS/XLV is almost nothing other than XFS has some advantage for zoned and the like.

Some of these really old file-systems are meant to be insanely feature rich yes you can count them on 1 hand and XFS is one them. Its also a horrible fact that a lot of feature rich file-systems when ported have lost huge volumes of their features.