Google Is Exploring Potentially Using Btrfs In Android

[GreatEmerald]

Try installing a linux distro on a slow flash drive with btrfs and use it for some time. Try the same but use f2fs then come back and tell me if you still want btrfs in a phone.

I've been using the Jolla phone for a while and I haven't seen any performance issues whatsoever.

There is also the question of which is going to be more reliable recovering from a dirty shutdown after a battery runs out of charge.

Btrfs, obviously.

Also don't forget that btrfs as been baking for so long and it isn't ready yet, if anything they could throw some more developers at f2fs and improve whatever needs improvement and fix whatever corner cases that need to be fixed.

It's ready for Jolla and SUSE. Pretty much all the snags have been ironed out by now.

[kreijack]

Try installing a linux distro on a slow flash drive with btrfs and use it for some time. Try the same but use f2fs then come back and tell me if you still want btrfs in a phone. There is also the question of which is going to be more reliable recovering from a dirty shutdown after a battery runs out of charge.

Also don't forget that btrfs as been baking for so long and it isn't ready yet, if anything they could throw some more developers at f2fs and improve whatever needs improvement and fix whatever corner cases that need to be fixed.

I use BTRFS from 2010 (or even before, but I don't remember). I used it even in a faulty hard disk (what sometime resets itself), and every time it was able to highlights the defect data (thank of the checksumming). So I am confident that in a standard setup it is definitely stable.
The problem born when you use some btrfs capabilities (like the multi RAID1/10/5/6) which aren't well tested/supported. In fact the biggest problem of BTRFS is that due to the HUGE capabilities set, it is very difficult to test all the possible setup. And even more to guarantee the performance in all workloads cases (big files, small files, databases mixed with different raid profiles, support for "hot" removing adding of disks.....).

I think that f2fs is great for small system; however I think that being a log structured filesystem and designed for flash memory, it should have the data protect by checksums.

[RussianNeuroMancer]

There is also the question of which is going to be more reliable recovering from a dirty shutdown after a battery runs out of charge.

I have few tablets based on Intel BayTrail and CherryTrail. Due to issues with Intel's drivers Linux tend to lockup on such devices from time to time. How frequently is depend on device and configuration, but there is combinations that lockup a lot. For example if I run Linux 4.11, Mesa 17.1.0 and Opera with disabled hardware rendering on my Dell Venue 8 Pro 5855 - this combination will be is mostly stable (I don't remember if this combination ever lockup but let's assume it's not 100% rock solid; that not really matter for my example anyway) however Linux 4.12 could lockup few times per hour with any latest Mesa and any Opera rendering settings. Because I use Linux 4.12 for testing of various patches I have no choice but get lockups very frequently, and hard reboot every time, and I think I done that hundred of times by now.

Filesystem on this Dell Venue 8 Pro 5855 is btrfs, no single fail so far. Well, if I continue to do hard reboots I guess eventually btrfs could fail to recover some day, but level of stability that I see right now is enough to not fail when "battery runs out of charge".

Obviously, there is reliable ways to make latest stable version of btrfs fail, something like scrub of re-balancing compressed RAID array should work I guess, but this is unrelated to mobile use-cases anyway.

[starshipeleven]

Try installing a linux distro on a slow flash drive with btrfs and use it for some time. Try the same but use f2fs then come back and tell me if you still want btrfs in a phone. There is also the question of which is going to be more reliable recovering from a dirty shutdown after a battery runs out of charge.

Note that flash in phones is significantly faster than crappy USB flash drives, and that Android just like other embedded systems lives in RAM and RAM-caches heavily to avoid slowdowns when reading/writing (some firmwares are more tuned than others).

The only thing that might be good to have is hardware acceleration for checksum calculation, I don't know if mobile devices have that baked in, most NAS devices have it because it's the same checksum algorithms they use for mdadm RAID5/6.

[starshipeleven]

The OS needs to reserve more space for itself than it thinks it needs regardless of partitioning scheme because no one wants to tell the user to delete pictures before their OS can install a security update.

AFAIK both Nexus and Apple stuff don't.
Tthey don't use separate /system partitions and if there isn't enough space they ask the user to delete stuff.

[linuxgeex]

How about snapshotting before android upgrades, compression, subvolumes, checksums, mirroring...
Remind me which of those ext4 has?

LVM can create COW snapshots of EXT4-formatted volumes, subvolumes can be emulated by using LVM to re-balance free space on the partition set. Mirroring is also LVM's domain. EXT4 can be compiled with optional per-file compression.

Now checksums, lol, BRTFS needs those pretty desperately to detect its failures. But I agree EXT4 could benefit from having checksums. Any filesystem that works on removable media is naive at best to not have both journalling and checksums.

Ideally, using cryptographically-secure checksums of data to detect block duplication and share those blocks, would be welcome in addition to compression.

While we're talking about ideal, having the kernel NOT dump the filesystem buffers automatically when a device is disconnected, but instead waiting for a few minutes or even seconds for the device to be reconnected, and/or allowing the UI an opportunity to ask the user to reconnect it so that the filesystem can finish syncing and unmounting, would be a very welcome addition to the Kernel VFS/Block layers.

[linuxgeex]

The only thing that might be good to have is hardware acceleration for checksum calculation, I don't know if mobile devices have that baked in, most NAS devices have it because it's the same checksum algorithms they use for mdadm RAID5/6.

"Most mobile devices" - we're talking about devices with ARM chips, which have hardware-accelerated cryptography and SIMD which can be used for strong or weak accelerated checksums. Also bear in mind that we're talking about future devices, not today's devices, because even if We The People came up with the most compelling arguments in history to change the filesystems currently in use, you can be certain that it would not be realised and delivered by OTA to more than 1% of today's devices. So I have less than no worries about future devices being able to keep up with checksums at the rate mobile storage devices can feed the vast number of cores those devices will have, lol...

I think most of us are missing the point though. A filesystem is garbage in garbage out, and a storage device is garbage in garbage out, but with a vastly higher chance of failure than ext4 itself. However the odds of a storage device dumping bad blocks back to the filesystem are getting vanishingly small. The block devices have layers of checksums, error correction, and redundancy (which vastly exceed the quality of BTRFS's checksums, but more about that later), and when they exhaust all those avenues (which takes time) and can't respond with good data, they don't respond with junk. They respond with a failure condition. Then the kernel block layer tries again a few times (which takes even more time and is why worn flash device performance drops off and why TLC drives are slower than MLC and MLC is slower than SLC) so don't kid yourselves that your files are not already protected by checksums even if EXT4 doesn't checksum them as well.

There's four cases where having the filesystem checksum the data matters:
1) the link to the block device is noisey, ie loose cable, bad motherboard, etc, and that is corrupting data.
2) powerfail during write, so then the filesystem needs to know whether the data was written correctly when it starts back up.
3) the operator triggers a hard reset of the storage device mid-write.
4) the block device returns crap when it can't read the data, without giving an error.

1 is common in Mobile. 2 used to be common in Mobile but these days the OS is reserving a significant portion of the battery to guarantee safe shutdown and to extend the battery life by preventing over-discharge. 3, well then they deserve it. 4 basically hasn't happened since the days of MFM/RLL harddrives apart from some of the very early super-crappy thumb drives and flash media with exceptionally poor checksums...

Poor checksums. Like CRC32. BTRFS currently uses CRC32, but has allocated room for a 256-bit checksum.

So how about let's rant about making BTRFS start using the bits it has allocated for checksums, to use a checksum stronger than the cheap hardware it's shepherding for us, so that it will actually be an improvement over no checksum at all? lol.

[starshipeleven]

Poor checksums. Like CRC32. BTRFS currently uses CRC32, but has allocated room for a 256-bit checksum.

And that is poor in what sense? Can I point out that is used by Ethernet, Sata, gzip, bzip2, png and a ton of others?

Can I also point out that Btrfs is using CRC32c, like ext4, Ceph and iSCSI?

Data integrity isn't encryption, they can use cheesier algos as data integrity checks are done once (or very few times anyway) per block/packet/whatever and if failed the data is assumed bad and discarded.
It's like using a short password, a 4-letter password is perfectly safe if after 2 tries your account goes in lockdown and does not accept any more tries.

Encryption algorithms have to hold infinite attacks for infinite time to be deemed secure. That's a bit different.

https://en.wikipedia.org/wiki/Cyclic_redundancy_check#Standards_and_common_use

[kreijack]

Now checksums, lol, BRTFS needs those pretty desperately to detect its failures.

It is not true. The checksum are computed as last step before sending the data on the disk. So even in case of bug, the data (wrong) would have its checksum computed on the wrong data.

[kreijack]

The only thing that might be good to have is hardware acceleration for checksum calculation, I don't know if mobile devices have that baked in, most NAS devices have it because it's the same checksum algorithms they use for mdadm RAID5/6.

The checksum algorithms are different from the ones used in the RAID5/6. I.e. a checksum could be crc32 or md5 or sha...., instead the parity in raid5 is a simple xor between the blocks.