ZFS On Linux Is Now Set For "Wide Scale Deployment"

[kernelOfTruth]

not exactly sure what's going on here:

I created the pool with sub-pools/volumes and copied already some data to it (around 800 GB), exported it

booted into windows


now came back to linux and wanted to import it:

zpool import WD30EFRX
cannot import 'WD30EFRX': one or more devices is currently unavailable

zpool import
pool: WD30EFRX
id: 6937543019016739157
state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
config:

WD30EFRX FAULTED corrupted data
WDred_zfs UNAVAIL corrupted data

these are pools with compress=lz4 set

I reproducibly got the same error when creating a new pool and after exporting trying to re-import it


any ideas ?



this is on an GPT partition table -> partition

might that be the reason ?


edit:


the same happens when using lzjb

[kernelOfTruth]

CPU utilization will likely increase by a few percentage points. However, I cannot speculate on what the effect will be on battery life. ARC could improve battery life while the periodic transaction commit (every 5 seconds) could harm battery life. If I were you, I would put everything on ZFS, rather than just /home.

With that said, you should consider LZ4 compression. It has a few properties that make it more appealing. One is that LZ4's throughput is an order of magnitude greater than gzip, which saves CPU time. Even more appealing is that LZ4 is extremely quick at identifying incompressible data. This property saves even more CPU time. As far as mobile use are concerned, these properties should translate into power savings in comparison to the situation where you use gzip. If you are interested in reading about LZ4, the following links are rather informative:

https://extrememoderate.wordpress.com/2011/08/

http://denisy.dyndns.org/lzo_vs_lzjb/

http://wiki.illumos.org/display/illumos/LZ4+Compression

https://code.google.com/p/lz4/

There are a few things to keep in mind when looking at those links:

1. The first link involves LZ4 r11. The version that ZFS imported is r67 (+/- 2, I forget the exact revision that was imported) and LZ4 has seen plenty of improvements since r11. Of particular interest is the time spent detecting incompressible data.
2. The second link compares LZO with LZ4. LZO is considered to be quick, but LZ4 beats it in every metric.
3. The third link is Illumos' writeup on LZ4, which compares it to ZFS' lzjb. lzjb was invented to obtain fair compression, have high throughput and detect incompressible data quickly. Those metrics are considered desireable for use in filesystems. LZ4 does so much better than lzjb in all of them that the Illumos developers initially thought that it was too good to be true when it was first discussed on their mailing list.
4. The fourth link is the LZ4 project page, which has a chart comparing the throughput and compression ratio of LZ4 to other compression algorithms. Of particular interest it that it shows that LZ4 has an average compression ratio of 2.1 while gzip -1 has an average compression ratio of 2.7, so you do not really pay much in terms of space for the benefits of LZ4

thanks a lot ryao !

that's a more in-depth answer than I had anticipated


I'm currently trying out ZFS/ZOL with lz4-algorithm on one of my backup disks and it looks good

I see one issue which hinders me from using lz4 on my laptop / on the root partition: there's no liveCDs with ZOL available that support the LZ4 algorithm

in case things go wrong I would have no access to my data at all if all partitions were using the LZ4 compression algorithm - am I correct ?

[uid313]

It's not a particularly simple problem to solve, for a number of reasons. One is that the whole installer image needs to fit in memory, so you add a lot to resource requirements if you need to compile from source (ie, the way the kernel mods are currently installed via DKMS).

ZFS is unsuitable for low memory systems anyways.
ZFS is intended for 64-bit systems with 4 GB RAM or more.

[pdffs]

It would be good if the linux installers would include some sort of zfs support.
You install the zfs package on the running install media, then you are able to partition and install to zfs. This way the zfs is not provided in the install media and no patent issues arise.

I was trying to do this with a fedora 18 live install but it's not very straight forward.

It's not a particularly simple problem to solve, for a number of reasons. One is that the whole installer image needs to fit in memory, so you add a lot to resource requirements if you need to compile from source (ie, the way the kernel mods are currently installed via DKMS). The only way to avoid that would be to precompile binaries for all the installers you want to support, then serve them via HTTP or something for the installer to download. The problem with that is, what happens when you don't have a net connection at install time? It's also pretty inefficient, since you need to serve the bins up every time an install occurs, rather than just once when the installer is downloaded.

An alternative to serving via HTTP might be to offer some sort of ability to use supplementary media, like a USB stick, to load the binaries but you've still got to compile those binaries to match every installer version you want to support (and add those media hooks to all the relevant installers). That's a fairly significant (and ongoing) task, particularly when installation media may get point releases that require new bins.

If you have the memory, using a livecd should already give you a moderately simple path to doing an initial install onto ZFS, since you should be able to boot to the live environment, install the packages as normal, prepare the disks as ZFS and perform the install onto them, then chroot into the install and perform any final configuration (here's a link for HOWTO install Ubuntu on ZFS root, haven't read it in a while, but from memory that was the gist of it).

[Samu23el]

The community should be able to help you with recovery.

[mo0n_sniper]

It would be good if the linux installers would include some sort of zfs support.
You install the zfs package on the running install media, then you are able to partition and install to zfs. This way the zfs is not provided in the install media and no patent issues arise.

I was trying to do this with a fedora 18 live install but it's not very straight forward.

[Sergio]

I would love to see Linux adopting ZFS as 'standard'; I though free/open source was all about merithocracy...

[ryao]

@Michael: Can you please do some benchmarks with the new zfs code. I would be interested to see the performance between ext4/btrfs/zfs.
Also if you could include OpenSolaris for a native zfs comparison that would be peaches

It would be better to use OmniOS in place of Open Solaris.

[mo0n_sniper]

Benchmarks

@Michael: Can you please do some benchmarks with the new zfs code. I would be interested to see the performance between ext4/btrfs/zfs.
Also if you could include OpenSolaris for a native zfs comparison that would be peaches

[pdffs]

Yeah Linux is great step for the technology and i appreciate those guy who make the Linux..How much money is required to get the license of the Linux?

All of the money.