Paragon Sends Out Updated NTFS Driver They Want To Mainline For The Linux Kernel

[mifritscher]

Regarding partition tools: I would love a "real" successor to Parititionmagic. parted had removed almost all filesystem code, so most operations are very basic. Not even efficient move operations, but also no conversions, merge/split of partitions with data etc. Partition Magic allowed almost everything one would dream to do with partitions. And it was very reliable.

A starter would be a collections of libraries (one per filesystem) which can
* check the file system
* do a full file+metadata <-> sector mapping (+ reverse operation: get free sector regions)
* move files to given sector regions
* add files by name and sector list
* needed offset+alignment for new files
* remove files
* change start + end of a file system by only moving metadata (+ a "dry run" which get the statically needed sector regions for metadata if any (super block, FAT etc.)
* create filesystem, with a dry run to get the statically needed sector regions
* align metadata to a new physical/logical/cluster sector size + change sector size (data must be moved beforehand) (physical: only optimization, logical: the new size is strict, cluster: can be greater than a sector (e.g. for fat))
* change properties from a filesystem, directory and file (at least at the first version: no abstraction needed)

All write operation should support a list of "forbidden" sector regions to e.g. ease conversions or merge operations to avoid overwriting data which aren't bound to the filesystem yet (ie: not added to its file structure). For filesystem creation: static metadata could be temporary redirected to other sectors to ease conversions - with a function to move them to the final destination.

This would allow the most needed extended operations, leaving as much as possible work in the "higher level" (e.g. allocation/move strategies).

Version 2 could support "versioned files" - ie. snapshots. Things like datasets (from zfs) could be seen as an additional directory layer at the API side (at least at the first version), zvols could be handled as files. Datasets could be an additional hierarchy type between the whole filesystem and directory structures as well or later.

Version 3 could be handling of filesystems which have more than one continuous sector regions (jbod, raidx etc.).

Version 4 would an extension to the driver so some operations could be done online ;-) For this, another function would be useful:
* reserve a sector region - permanent (if possible) or temporal (until umount)

[ALRBP]

Most existing smart TVs support fat and ntfs only. exFat is still relatively new and its support leaves a lot to be desired.

exFAT is supported by a lot of Android devices (maybe all after mainlining) while NTFS is not. This is why I stopped using NTFS for my "multi-system" USB key, in favor of exFAT. That said, I still have a FAT32 "all terrain" key ; FAT32 is still the only thing you are sure to be working everywhere and will probably remain for long, until all non-exFAT supporting devices are gone (notably, Linux-based pre-exFAT mainlining systems, since a lot of them do not include FUSE-based exFAT support). NTFS is too complicated and will never be an "all terrain" FS, unlike exFAT. While NTFS was previously the only way to share >4 GB files with Windows, and thus adopted by some old TVs, it has been obsolesced by exFAT for this application.

I personally care even less about NTFS mainlining than exFAT mainlining. The main advantage of exFAT mainlining, with MS removing the patent threat, is that now, Linux based (notably Android) devices are more likely to include exFAT support, without fearing a trial from MS if they do not pay a license. For NTFS, well, all modern systems supporting it will also support exFAT, some systems will likely only support exFAT (Linux post-exFAT but pre-NTFS or just devices wanting to support no more than SD specification, with mass storage devices formatted in exFAT by default) and NTFS-3G is sufficient for Linux desktop anyway (especially if you just want to share with your Stone Age smart TV that you should anyway pair with a modern (far cheaper than a new TV) Android TV box if you want to read modern codecs, like H265 or AV1).

[starshipeleven]

Ok, but maybe those filesystem specific tools like ntfstools could use this driver in the future when it moves data around the drive like when you resize a NTFS partition and it says that this operation could take a long time because it relocates data.

Afaik for most filesystems the tools are working at a lower level and moving blocks manually and then adjusting the index tables to reflect the new locations, while the partition is unmounted and the driver isn't active.

Afaik the only filesystems where the filesystem driver is involved in these operations are the next-gen filesystems like ZFS and Btrfs, where you can issue requests to relocate blocks to the driver and it will actually do the leg work while the filesystem is still online and usable by other applications/users. (used by balance, scrub, remove device, and shrink partition size)

[starshipeleven]

Regarding partition tools: I would love a "real" successor to Parititionmagic. parted had removed almost all filesystem code, so most operations are very basic. Not even efficient move operations, but also no conversions, merge/split of partitions with data etc. Partition Magic allowed almost everything one would dream to do with partitions. And it was very reliable.

A starter would be a collections of libraries (one per filesystem) which can
* check the file system
* do a full file+metadata <-> sector mapping (+ reverse operation: get free sector regions)
* move files to given sector regions
* add files by name and sector list
* needed offset+alignment for new files
* remove files
* change start + end of a file system by only moving metadata (+ a "dry run" which get the statically needed sector regions for metadata if any (super block, FAT etc.)
* create filesystem, with a dry run to get the statically needed sector regions
* align metadata to a new physical/logical/cluster sector size + change sector size (data must be moved beforehand) (physical: only optimization, logical: the new size is strict, cluster: can be greater than a sector (e.g. for fat))
* change properties from a filesystem, directory and file (at least at the first version: no abstraction needed)

All write operation should support a list of "forbidden" sector regions to e.g. ease conversions or merge operations to avoid overwriting data which aren't bound to the filesystem yet (ie: not added to its file structure). For filesystem creation: static metadata could be temporary redirected to other sectors to ease conversions - with a function to move them to the final destination.

This would allow the most needed extended operations, leaving as much as possible work in the "higher level" (e.g. allocation/move strategies).

Version 2 could support "versioned files" - ie. snapshots. Things like datasets (from zfs) could be seen as an additional directory layer at the API side (at least at the first version), zvols could be handled as files. Datasets could be an additional hierarchy type between the whole filesystem and directory structures as well or later.

Version 3 could be handling of filesystems which have more than one continuous sector regions (jbod, raidx etc.).

Version 4 would an extension to the driver so some operations could be done online ;-) For this, another function would be useful:
* reserve a sector region - permanent (if possible) or temporal (until umount)

sounds like a good weekend project for the next few decades

[drSeehas]

... NTFS was previously the only way to share >4 GB files with Windows, ...

???
What about UDF?

[sinetek]

Paragon wrote they would release the fsck and mkfs utility at a later point when they've been over it more thoroughly for a gpl release.
I want to see this merged in, I think Paragon is doing a great thing here and the code's good.

[mishakov]

exFAT is supported by a lot of Android devices (maybe all after mainlining) while NTFS is not. This is why I stopped using NTFS for my "multi-system" USB key, in favor of exFAT. That said, I still have a FAT32 "all terrain" key ; FAT32 is still the only thing you are sure to be working everywhere and will probably remain for long, until all non-exFAT supporting devices are gone (notably, Linux-based pre-exFAT mainlining systems, since a lot of them do not include FUSE-based exFAT support). NTFS is too complicated and will never be an "all terrain" FS, unlike exFAT. While NTFS was previously the only way to share >4 GB files with Windows, and thus adopted by some old TVs, it has been obsolesced by exFAT for this application.

I personally care even less about NTFS mainlining than exFAT mainlining. The main advantage of exFAT mainlining, with MS removing the patent threat, is that now, Linux based (notably Android) devices are more likely to include exFAT support, without fearing a trial from MS if they do not pay a license. For NTFS, well, all modern systems supporting it will also support exFAT, some systems will likely only support exFAT (Linux post-exFAT but pre-NTFS or just devices wanting to support no more than SD specification, with mass storage devices formatted in exFAT by default) and NTFS-3G is sufficient for Linux desktop anyway (especially if you just want to share with your Stone Age smart TV that you should anyway pair with a modern (far cheaper than a new TV) Android TV box if you want to read modern codecs, like H265 or AV1).

I had actually found exFAT (or any FAT*) filesystem(s) to be terrible for file sharing between Linux and Windows due to the lack of journaling, case sensitivity, symlinks, compression and ACLs (in Windows) and was much less excited about exFAT mainlining than about NTFS now. You would never know what would break when using it on Linux (because of the case-insensitivity), and after recently giving it a try, I had reformated my external drives back to NTFS (and BTRFS/XFS on my Linux NAS) after one of them had exFAT go irreparably corrupt (something I've never seen on journaled filesystems, but have (unsurprisingly) witnessed many times on FAT*). I have never seen a device which would support exFAT but can't at least read NTFS, but a lot that only supported the latter and I'm not expecting new devices to give up supporting NTFS due to its wide adoption. And with these patches, it will now perform as good as other Linux filesystems and fully support journaling, IMO possibly making it a perfect choice for the drives to mostly be used on Linux systems, but which could also be easily accessed on Windows (without fiddling with drivers) and read by pretty much any other OS/device. And although ntfs-3g might be sufficient for some, it is much slower, especially on low-end linux machines (e.g. Raspberry Pis) and isn't very reliable due to it not supporting journalling.

[starshipeleven]

???
What about UDF?

lack of fsck tools, funky handling if the partition is bigger than 1TB

[bemerk]

By the way, is anyone working to mainline support for Apple's APFS in the Linux kernel?
I know there was some code on GitHub for APFS support in the Linux kernel, as well as a FUSE driver.

The github stuff is far from ready as it is a spare time reverse engineering effort.
Paragon could speed that up a lot

[uid313]

The github stuff is far from ready as it is a spare time reverse engineering effort.
Paragon could speed that up a lot

My impression is that the stuff on GitHub have accomplished read-only support for APFS. But maybe it was just read-only without decompression of transparent compression. I don't know if Paragon is involved in APFS, I just know they do NTFS.