OpenZFS Is Still Battling A Data Corruption Issue

[onlyLinuxLuvUBack]

until they rewrite all of openzfs in rust, make 3 partitions that go into a bar: ext4, openzfs, and btrfs.

[Mario Junior]

This is a CVE now

https://www.cve.org/CVERecord?id=CVE-2023-49298

[stiiixy]

Tested backups, with a process to restore the data and to validate. If you don't test restoring and validating your backup you do not actually have a backup. It is unfortunately true that there are those that learn that lesson the hard way.

This. He's upvoted for a reason.

[stiiixy]

So true.

Well, at least it happened 3+ decades ago, and I didn't lose everything, just some important (to me) C code I had worked hard to develop. Luckily, at that time we, developers, used to write down the important aspects of whatever on paper, and, as so, it was a matter of painful code rewrite time to put almost everything back, almost.

Hard lessons leave a persistent imprint.

Weirdly, I found (not as a dev) that the written work provided a hidden insight when 'revised' after such events.

That insights was the hard lesson, but you come out with better results for having bathed in fire, so to speak.

[miquels]

For sure! Here is my OpenZFS PPA for recent ubuntu versions (not LTS) with the patch from https://github.com/openzfs/zfs/pull/15571 for this issue applied: https://launchpad.net/~satadru-umich...-experimental/

In the FreeBSD advisory in the article, they mention a workaround with sysctl settings. If you don't want to recompile the linux kernel, this is the same workaround for Linux:

echo "# https://lists.freebsd.org/archives/freebsd-stable/2023-November/001726.html" >> /etc/modprobe.d/zfs.conf
echo "options zfs_dmu_offset_next_sync=0" >> /etc/modprobe.d.zfs.conf
echo 0 > /sys/module/zfs/parameters/fs_dmu_offset_next_sync

[mb_q]

The overriding, and surprising, issue for me after the last week is that no one developing ZFS appears to know how it works. And this is the exact same problem I observed with the BTRFS developers.

This is in part because POSIX FS consistency model is "whatever" and disk hardware consistency models are generally "go f* yourself". You have to be able to reason about systems to have a clean design, otherwise you accumulate quirk-fixes and everything approaches pure guess. Vide classic arguments around Netscape rewrite.

[ultimA]

until they rewrite all of openzfs in rust, make 3 partitions that go into a bar: ext4, openzfs, and btrfs.

Oh for f**k's sake. No, Rust would not have helped this bug. This was a logic error. Not memory related. Not even a race condition. Yes, even though it received a CVE and was deemed security-relevant. When will you Rust-maniacs finally get that Rust will not lead us into a world with no vulnerabilities, and much less into one without bugs?

Yes, Rust is more secure. But stop pretending (and demanding) that everything should be rewritten in Rust, this is stupid.

[_ReD_]

That's even worse though, because it was recommended as production, then data started disappearing and then they were like "whoa..whoa...JK...don't use that".

I call bullshit! Put up a link to the recommendation or shut up.

[rrveex]

Oh for f**k's sake. No, Rust would not have helped this bug. This was a logic error. Not memory related. Not even a race condition. Yes, even though it received a CVE and was deemed security-relevant. When will you Rust-maniacs finally get that Rust will not lead us into a world with no vulnerabilities, and much less into one without bugs?

Yes, Rust is more secure. But stop pretending (and demanding) that everything should be rewritten in Rust, this is stupid.

I think OP was joking. Isn't nowadays "rewrite it in Rust" always a joke?!

[Developer12]

Disappointed to see Michael double-dipping on this news. Post an article about the bug, wait a few days, post another article with basically no change to the situation.

It hasn't been fixed, it's still being root-caused, but he just can't seem to resist the traffic spike.