Btrfs Seems To Finally Have Failed Me On A Production System

[arokh]

Why would you upgrade to an RC kernel on a "production" system? I've been running btrfs on my Arch HTPC for about two years, the only problem I ever had was after upgrading the kernel at one point. Had to run the zero log command from a rescue disk and then btrfsck. Everything running smoothly after that.

I'm even using snapshots quite heavily, and it's saved me countless times when I messed things up. Just boot back into a previous snapshot and to a working state.

[Michael]

That systems production task is just benchmarking the latest Git kernel every morning.

Why would you upgrade to an RC kernel on a "production" system? I've been running btrfs on my Arch HTPC for about two years, the only problem I ever had was after upgrading the kernel at one point. Had to run the zero log command from a rescue disk and then btrfsck. Everything running smoothly after that.

I'm even using snapshots quite heavily, and it's saved me countless times when I messed things up. Just boot back into a previous snapshot and to a working state.

[SystemCrasher]

That was 20 years ago. Not since then...

Wrong! Even latest Win 8 could easily hit BSOD if ntfs driver faced corrupted file system. Should there happen to be bad sector on drive - it goes really nuts! Corrupted filesystem structures? Get this cool BSOD. Now! And while ntfs driver BSODs, scandisk/chkdisk often fails to find any issues at all, giving a lot of lulz to owner who is completely unable to mount troublesome volume in windows. Windows just BSODs on mount attempt and ... you can't access your data at all using windows. Needless to say, usual users admit EPIC FAIL in such situation and forced to seek help of some gurus around. While I'm not hardcore data recovery guru, I sometimes bother myself to be "in-situ data recovery specialist" and managed to recover data in about 5 such dumbass cases, by simply mounting "faulty" volume under NTFS-3G fuse driver in Linux and copying all or almost all data to another drive, so drive could be re-formatted and is getting back into order. This simple, cheap and dirty trick saved some users a lot of woes :P. And I really do not get why scandisk and chkdisk fails to find errors while driver dares to crashes on mount. That's not what I would call good filesystem toolkit.

Actually, M$ haves surprisingly low code quality: as long as things are not ideal, you can expect worst. Windows still sometimes BSODs when you just insert flash drive, failing to handle debounce at initial connection properly and likely facing some silly race condition. Despite claims they "rewrote whole kernel", this bug persists for about 15 years or so, he-he-he. So it seems they "rewrite" their kernels using copy and paste (why its ok on its own, bold claims and lies about "rewrite" are not ok!).

As for btrfs...
1) fsck mostly a stub to keep boot process happy since btrfs on its own not really supposed to face fatal faults unless underlying hardware failed, etc. It is similar to ZFS in this regard. I guess best idea to boot from livecd (looking on errors it seems your kernel modules gone nuts so I wouldn't expect such OS to work properly). And then use "btrfs check" I guess, if it refuses to mount. Or if it mounts, btrfs scrub could be an option. I suspect faulty kernels maybe wrote some garbage data to disk or so. Yet it's not a good reason to hang indefinitely - filesystem should deal with faulty data and at least return errors and fail requested operations, returning I/O error or so.

For more details on checking btrfs: https://btrfs.wiki.kernel.org/index.php/Btrfsck

Note: it is better to use really recent kernel and tools for recovery. Bugs in FS tools and kernel are not fun, after all :P.

2) If everything has failed BUT you still need to get some valuable data back, you can try "btrfs restore" - a non-destructive attempt to read data and salvage what is actually readable. See https://btrfs.wiki.kernel.org/index.php/Restore for more details.

[profoundWHALE]

Wrong! Even latest Win 8 could easily hit BSOD if ntfs driver faced corrupted file system. Should there happen to be bad sector on drive - it goes really nuts! Corrupted filesystem structures? Get this cool BSOD. Now! And while ntfs driver BSODs, scandisk/chkdisk often fails to find any issues at all, giving a lot of lulz to owner who is completely unable to mount troublesome volume in windows. Windows just BSODs on mount attempt and ... you can't access your data at all using windows.

Their stupid fast boot causes so many headaches for me. If anything happens to the system while trying to shut down, it's essentially, "screw you!" and you have to reinstall. Of course, you might think that they've either ironed that out or that it doesn't happen often. It's happened to me around 3 times with Windows 8-8.1 and 2 times already with Windows 10.

[bulletxt]

Michael, if you continue loosing money you should eventually stop.

This probably ins' t a good business and I don't see this changing in the future.
Make the entire website pro, if people think yours is a good service they will eventuslly pay. Otherwise amen.

[torsionbar28]

Could also be a failing SSD of course: Check SMART data and try mounting the filesystem(s) on the drive *before* pointing btrfs-tools at the suspect partition?

That was my first thought as well. If multiple systems are running the exact same code, and this particular system is the only one experiencing an issue, I would first validate the hardware before assuming it's a software bug. The SMART data should hold some clues if it's a hardware failure.

[ferry]

Seems like the deadlock issue that hit kernel 3.19 stable release users:

https://btrfs.wiki.kernel.org/index.php/Gotchas

Try use btrfs-zero-log with a older/higher kernel.

One of my Fedora box got hit by this, 4 others remained fine before patched kernel push to update repo.

I had problems 3x recently on 2 systems, where system freezes during dpkg (during synaptic updates). After this, during reboot the kernel crashes letting (during btrfs scan) me fall back to busybox. But then nothing you can do. I booted with a Live USB, but this crashed as well as it scans btrfs as too. So black listing the btrfs kernel module of the Live USB got me to boot, then modeprobe -i btrfs, then btrfs-zero-log and finally btrfs check fixed it.

On the topic of ntfs: hah!. In 30 years never been able to recover a broken ntfs other than by booting a Linux Live CD/USB and copying the accessible files. I heard this is even how they do it at Microsoft.

[fsproc]

ruhe ,

[chrisb]

That was 20 years ago. Not since then...

Windows Home Server NTFS corruption bug. That was shipped with Windows Home Server and it took MS 7 months to fix it. In the meantime there wasn't even a workaround - the only advice was to "have a backup copy of any important program files before you store these files on a system that is running Windows Home Server."

[DanL]

Hahaha btrfs is the future they say... hahahahahaha losers all the way. When was the last time ntfs did something like this? 30 years ago? Who are the amateurs working on btrfs, so I know never to use any of their products.

Could someone send this troll's account to the great bit bucket in the sky?