F2FS Preparing To Introduce New "Secure Erase" Functionality

[uid313]

F2FS is designed to not do small writes (Flash can only erase blocks in the order of 2Megabyte or more, so the worst thing you could do is writing the same byte multiple times), benefiting flash, but having drawbacks (obviously). The more powerful a FTL the more it will accumulate writes before sending it to the Flash, it will need meta-data to restore the "view" of the PC aswell. F2FS does some of this by design, not needing "emulation" and the overhead in meta information.
You can compensate the performance with a bigger and fatter FTL, but the meta information needs to end somewhere, causing more writes compared to F2FS. But then a big cache might mean some writes never end up on flash.

Interesting. I wonder what kind of file systems we will have in a decade or two and how they will operate.

[CommunityMember]

I know that F2FS is used on some smartphones, but is it only used on smartphones or is it used on laptops and desktops too?

For a specific use case I have used F2FS for USB sticks that were subject to constant power outages and heavy use(*), and the lifetime and recovery of the (as I recall from at the time, really cheap/crap) USB sticks went up significantly. As is typically the case, there are specific targeted points where any filesystem is better than others.


(*) yes, that is not a great idea, but the specific use case made it the better of a bunch of different poor choices.

[zyxxel]

someone uses it, it can be used.

SSDs used in desktop/laptops are running a FTL anyway as they must be able to run NTFS/Windows so the concept of "flash-friendly filesystem" is meaningless for such devices as is any performance difference with other normal filesystems.
None cares about f2fs features so far as it has no snapshots and volumes.
Everyone invested in server/desktop has already a puppy (btrfs, ZFS or Stratis)

You need to reconsider your view on "none cares". F2FS isn't aimed for server/desktop where you have a very intelligent memory controller hidden in the SSD. But the server/desktop isn't the only area where the world runs Linux - the huge majority of Linux installations are in embedded devices. For some strange reason, people in the embedded world cares about things you probably haven't even considered existed.

[starshipeleven]

You need to reconsider your view on "none cares".

I am specifically talking of desktop/laptop SSDs.

F2FS isn't aimed for server/desktop where you have a very intelligent memory controller hidden in the SSD

It was born for and it is most commonly used in Android devices. Outside of that, it's either missing features, or it is not as good as other filesystems.
For example it can compress, but can it compress better than squashfs? Not really. And for a static firmware image you are 100% not caring about the fact that squashfs is read only.

the huge majority of Linux installations are in embedded devices.

embedded devices use raw NAND or NOR chips, which are not block devices so can't use F2FS.

Decent ones are using UBI/UBIFS, or jffs2 for read-write partitions and squashfs for read-only partitions (most of the flash space usually)

Stuff with eMMC is more in the industrial sector or for expensive appliances for businness (firewall and such)

[discordian]

I am specifically talking of desktop/laptop SSDs.

It was born for and it is most commonly used in Android devices. Outside of that, it's either missing features, or it is not as good as other filesystems.

Its pretty much on par with ext4, or what are the downsides here?

For example it can compress, but can it compress better than squashfs? Not really. And for a static firmware image you are 100% not caring about the fact that squashfs is read only.

A tar.xz can compress even better. Apples and oranges really.

embedded devices use raw NAND or NOR chips, which are not block devices so can't use F2FS.

I haven't dealt with much NAND devices anymore, seems like eMMC is pretty much everywhere nowadays (thank god). Atleast everywhere you can run Linux on top.

Decent ones are using UBI/UBIFS, or jffs2 for read-write partitions and squashfs for read-only partitions (most of the flash space usually)

Nand has a really funny characteristic, that even when just reading, neighbouring cells will lose its charge, and bit errors can always occur. You cant use squashfs on raw NAND.

Stuff with eMMC is more in the industrial sector or for expensive appliances for businness (firewall and such)

What sectors do you think of where that is not the case.

[starshipeleven]

Its pretty much on par with ext4, or what are the downsides here?

On PC? Much less testing and much less experience in issues that can happen to it if compared to the other filesystems, so its fsck isn't as good in case you are in trouble, if compared to ext4/xfs.
If compared to btrfs/zfs it lacks features.

A tar.xz can compress even better. Apples and oranges really.

A tar.xz can't be mounted and read like a filesystem so you need to waste some RAM to make a ramdisk to decompress it into, and you may or may not have that space, squashfs is a filesystem, you can mount it and read the contents while it is still compressed.

I haven't dealt with much NAND devices anymore, seems like eMMC is pretty much everywhere nowadays (thank god). Atleast everywhere you can run Linux on top.

You are not making consumer stuff I guess.

Nand has a really funny characteristic, that even when just reading, neighbouring cells will lose its charge, and bit errors can always occur. You cant use squashfs on raw NAND.

I'll send a notice to Zyxel, Netgear, Asus, Synology and a whole list of consumer embedded device manufacturers that discordian said they can't do that.

I've also seen a mtd partition that was a tar.xz flashed raw on NAND, that was decompressed on boot into a RAMdisk (Zyxel NSA31x line of NAS products), how's that for bit errors?

Ah, and kernel image is still 100% flashed raw on NAND everywhere, no OEM is using a sane uboot that can load stuff from UBI, and in many cases it won't handle bad blocks in the kernel mtd partition either so if that part of the flash develops a bad block the device is fucked.

OpenWrt usually places the squashfs inside an UBI volume, so it's at least protected from bad sectors and such (bad sectors that can happen when doing a firmware upgrade, as that is the moment you are writing a new squashfs image).

What sectors do you think of where that is not the case.

Consumer embedded. Not smartphones/tablets of course, but IP cameras, the central control station for the cameras, NAS, network devices (routers/access points), more specific devices that can't just recycle smartphone hardware, and a whole lot of IoT stuff for consumers, like the "smart washing machines" that have wifi and can connect to a central server so you can start/stop it remotely (totally safe I swear).

[zyxxel]

I am specifically talking of desktop/laptop SSDs.

It was born for and it is most commonly used in Android devices. Outside of that, it's either missing features, or it is not as good as other filesystems.
For example it can compress, but can it compress better than squashfs? Not really. And for a static firmware image you are 100% not caring about the fact that squashfs is read only.

embedded devices use raw NAND or NOR chips, which are not block devices so can't use F2FS.

Decent ones are using UBI/UBIFS, or jffs2 for read-write partitions and squashfs for read-only partitions (most of the flash space usually)

Stuff with eMMC is more in the industrial sector or for expensive appliances for businness (firewall and such)

Some embedded devices are using raw NAND/NOR. But that is a simplification, because lots of embedded devices are likely to use multiple memory technologies. And eMMC is really quite common when designing embedded products running Linux - lots of the processors moves towards these solutions. If things aren't networked, it's likely to be a microcontroller, so no Linux and file system at all. When networked, it's likely to need encryption etc - so firmware updates etc becomes important.

UBIFS works reasonably well for raw flash - JFFS2 has a couple of significant issues, such as huge boot times if there are lots of files in the file system.

[starshipeleven]

lots of embedded devices are likely to use multiple memory technologies. And eMMC is really quite common when designing embedded products running Linux

and that's a block device with a decent FTL, so there is really not a lot of difference between F2FS and ext4 there.

[zyxxel]

and that's a block device with a decent FTL, so there is really not a lot of difference between F2FS and ext4 there.

Just that ext4 is notorius for failing to boot after hard power loss.

[discordian]

On PC? Much less testing and much less experience in issues that can happen to it if compared to the other filesystems, so its fsck isn't as good in case you are in trouble, if compared to ext4/xfs.

neither fsck nor extundelete ever helped me with ext4, f2fs aint as old as ext4 but its been in use since 2012.

If compared to btrfs/zfs it lacks features.

I never claimed different

A tar.xz can't be mounted and read like a filesystem so you need to waste some RAM to make a ramdisk to decompress it into, and you may or may not have that space, squashfs is a filesystem, you can mount it and read the contents while it is still compressed.

No shit, and a RO filesystem is a differnt ballpark as a read/write one.

You are not making consumer stuff I guess.

Industrial and realtime, hard + software. I deal with embedded CPUs running linux down to STM32 and AVR. No consumer stuff, yes.

I'll send a notice to Zyxel, Netgear, Asus, Synology and a whole list of consumer embedded device manufacturers that discordian said they can't do that.

NAND "read disturb" exists, if you have Explanation to why this is no issue?
is that really "raw" NAND or does it come with some proprietary blob, is it some SPI chips that atleast do FEC (NAND is bloody useless without), how are bad sectors handled.
I might not be in the consumer branch but I evaluated several propietary FTLs and NAND Chips (now nearly a decade back).

I've also seen a mtd partition that was a tar.xz flashed raw on NAND, that was decompressed on boot into a RAMdisk (Zyxel NSA31x line of NAS products), how's that for bit errors?

From my memory the rate of raw bit errors was around 1 in a million, good luck storing a few MB without FEC.

Ah, and kernel image is still 100% flashed raw on NAND everywhere, no OEM is using a sane uboot that can load stuff from UBI, and in many cases it won't handle bad blocks in the kernel mtd partition either so if that part of the flash develops a bad block the device is fucked.

Guess they saved 4$ by not using a NOR for the kernel. Sounds really fun.

OpenWrt usually places the squashfs inside an UBI volume, so it's at least protected from bad sectors and such (bad sectors that can happen when doing a firmware upgrade, as that is the moment you are writing a new squashfs image).

A bad sectors is one where there are more Errors than a FEC can correct, without FEC you cant reliably do anything (back to why I cant imagine someone using raw NAND).

Consumer embedded. Not smartphones/tablets of course, but IP cameras, the central control station for the cameras, NAS, network devices (routers/access points), more specific devices that can't just recycle smartphone hardware, and a whole lot of IoT stuff for consumers, like the "smart washing machines" that have wifi and can connect to a central server so you can start/stop it remotely (totally safe I swear).

You have some model numbers of the SOCs used?

A direct NAND IF is a thing of the past AFAIK, look for ex. at the i.MX283 that had multiple channels for NAND chips and you had to do FEC yourself. Might be that I am out of the loop on that, and SPI chips do that in HW?