Linux Gets A Proprietary, Read-Only ReFS File-System Driver

[pal666]

Can't we all just agree that whatever I install on my supposedly free Linux system is my own damn business?

as long as you install your own stuff

[pal666]

No, what Daktyl198 says is not a license violation.

yes, it is

You're free to link in proprietary code into your GPL'd kernel, as long as you do not redistribute it.

and as long as proprietary code's license allows it

[ssokolow]

No, what Daktyl198 says is not a license violation. You're free to link in proprietary code into your GPL'd kernel, as long as you do not redistribute it. GPL does never say anything about using software, only about distribution.

Yes, but the issue is whether it's legal for them to distribute a GPL-incompatible Linux kernel module to you. (ie. Whether the module is a derived work in the eyes of the law. If so, then the GPL applies to it.)

nVidia seems to be relying on the fact that, since their binary blob is shared between the Linux and Windows drivers, they can argue that it's legally derived from neither codebase.

In this case, does this blob have any utility outside of being linked against a GPLed project? If not, then the courts would probably rule it a violation of the kernel's license.

[DrYak]

Finally, for those wanting cross OS filesystem, there are one FS that everyone supports and it is always forgotten: UFS https://en.wikipedia.org/wiki/Universal_Disk_Format

Small nitpicking: it's called "UDF", UFS is an entirely different system.

This FS was build for DVD, specially DVD-RW, but have standard extensions for normal disk usage.

Micro-nitpicking: harddisk are also considered on the Plain build of UDF 1.0. It was designed from day 1 for harddisk (in addition to optical read/write media).
Though as mentioned by @ssokolow, Windows and Mac have contradictory and opposite approach to partitioning regarding UDF.
So ELJericho's script is mandatory for true multi-OS support.

This FS have only 2 problems {...} there are few tools to recover/repair it... but other than that, works very well.

UDF is a log structured filesystem (like F2FS, NILFS, similar to CoW filesystems like BTRFS and ZFS).
Each time you write something on the disk/flash drive, it is added as a new entry in the log. Data is never over-written.
(Eventually, on read/write media, the oldest entries will get garbage collected to claim free space once space is needed and these entries aren't relevant anymore).
Think of it as the successive burning sessions on a CD-R {except on a much smaller granularity than whole ISO9660 session.} :You don't overwrite the old content (on a write-once media like CD-R you just can't) instead you add new content that indicates modifications to the old content.

Older over-writing filesystem like old-school EXT2 or FAT16/FAT32/exFAT do over write data. If something goes wrong mid-writing (power outage, crash, or simply media corruption) you end up with content that is neither old correct data, nor what the new data should correctly look like.
You can screw your content, and you need special tools to try to make sense out of the mess and try recovering as much as possible.

(On more modern file system, such mid-writing corruption is partly avoided by keeping a journal: Journaling in EXT3, XFS, NTFS, ReiserFS, etc. and you can mitigiate corruption in some cases simply by replaying the journal instead of running fsck.)
(In case of log-structured (and somewhat CoW) filesystem: the whole filesystem IS the journal).

On the other hand, by definition of how log structured (and CoW - it's even in its name) work, you can NEVER corrupt old data by writing : you only write new data, never over write older.
So in case of mid-writing fail (like power-failure or crash) you just end up with 2 copies: a half baked new copy that can't be used, and the older copy that is still how you left it.
Thus filesystem checker aren't even that much needed.
- Some filesystems don't even bother with a fsck tool. That's the case of UDF.
- Other filesystems like BTRFS have one (written by the openSUSE people) but it comes with big warning that it should only be used as a LAST RESORT when all other "classical approaches to CoW recovery" have utterly failed.

Again, if you take my comparison with multi-session CD-R :
- if one burning session fails (power failure, crash) or gets damage (CD reflective surface/dyes scratches), you end up with:
- a new half-burned session that is useless and unreadable
- all the older session(s) that are still around and that you can still use as-is.
the CD-R is still usable, you just can't get anything out of the latest session (well, it also won't work on older pure CD-ROM / CD-Audio drives that don't support multisessions, but that's a different story)

So usually, you recover data on log-structured and CoW system simply by mounting them.
(UDF will try to mount the latest log entry that is usable. BTRFS offers tons of options to select older copies, or be more tolerant about "usable" entries).

So UDF won't suffer as much as EXT2 / FAT32 / etc. by lacking a fsck tools.


On the other hand log-structured and CoW systems have a major drawback for users who aren't used to them:
The way "free space" works is completely bonkers.
(see all the "enospace" on BTRFS, although "df" doesn't report 100% usage).

Among other: *deleting files* is an operation that unintuitively DOES NOT immediately free space, but actually *requires* free space initially.
(you need to add a new entry in the UDF log, that obsoletes the older log entries with the deleted file before those entries could be garbage collected to claim the space).

Old linux kernels used to have a bug where the UDF driver could manage to paint itself into a corner where there isn't any space left on the device to the point that the driver can't event delete files to make space.
Luckily, this bug is fixed in modern kernels.

Morality: if you're not experienced manipulating log-structured or CoW filesystem, it's better to always leave plenty of free space on them and never fill them all the way up to exactly 100.0%

(and in the case of btrfs: use a linux distro that features tools to automatically keep the filesystem groomed for you : like OpenSUSE's btrfs-maintenance-tools, or Jolla SailfishOS' btrfs-balancer).


Log structured (and CoW) filesystems come with an indirect advantage: they are *MUCH* gentler on flash (and optical-rewriteable) media that has lower earse cycles.
- There's a reason why log-structure was picked up for UDF as it was planned for CD/DVD
- The "F2" of F2FS stands for "Flash Friendly" and log-structured is the reason why.
(Also partly the reason why BTRFS was picked up for Jolla's first phone's internal flash)

So in short UDF on a flash media (USB stick) is a *very good* idea (as long as you don't fill it all the way up to 100.0%)

[higuita]

Small nitpicking: it's called "UDF", UFS is an entirely different system.

Oopps, you are correct!

Micro-nitpicking: harddisk are also considered on the Plain build of UDF 1.0. It was designed from day 1 for harddisk (in addition to optical read/write media).
Though as mentioned by @ssokolow, Windows and Mac have contradictory and opposite approach to partitioning regarding UDF.
So ELJericho's script is mandatory for true multi-OS support.

Also true. UDF 2.5 and 2.6 is mostly blue-ray and mostly no one use then, mkudffs do not even support then, the 2.01 is the max
When i used it, i manually formated with the 2.01 format and created the fake mbr, but i forgot about that, so sorry to not reference that... but the script is perfect!

So UDF won't suffer as much as EXT2 / FAT32 / etc. by lacking a fsck tools.

Yes, UDF corrupt much less, but fsck (and other recovery tools) are need also for fixing bugs or wrong behavior (corrupt kernel writing trash/wrong place, bad physical storage and recover a image). A resilient format may always be corrupted in some way. For 99.9% of the people this is not a issue, for some people (specially long term archiving) this may be a issue. That is why i warn about it.

Anyway, using UDF for cross-platform fs is way better than the other alternatives

[chithanh]

I have experimented with sharing a UDF partition on a Windows/Linux dual-boot setup some years back and my experience was not so good.

• Creating UDF with mkudffs may lead to Windows being unable to write to the filesystem.
• Mucking with filesystem permissions on Linux side caused files to become inaccessible on Windows.
• UDF sparse file handling on Linux was pretty bad, I encountered kernel oopses and data corruption. And no working filesystem repair tool.
• Sometimes, files created on Linux would not be visible at all on Windows.
• No journaling would be supported either, as that feature came after UDF 2.01.

[DrYak]

Yes, UDF corrupt much less, but fsck (and other recovery tools) are need also for fixing bugs or wrong behavior (corrupt kernel writing trash/wrong place, bad physical storage and recover a image). A resilient format may always be corrupted in some way. For 99.9% of the people this is not a issue, for some people (specially long term archiving) this may be a issue.

(Note that I didn't say that fsck on CoW and log-structured FS is useless, only it's less a requirement than on over-writing systems).

If you start to consider data being eaten by bugs, instead of counting on a fsck tool (which still might be handy), I would *strongly* suggest concentrating one's effort on a proper backup solution.

Like sending data (e.g.: using rsync) over to a server where you keep history of older version (e.g.: using a snapshotting FS like btrfs or zfs, and a tool like snapper).

If you fear your data being corrupted by bugs, keeping additional copies of the data elsewhere helps.

I have experimented with sharing a UDF partition on a Windows/Linux dual-boot setup some years back and my experience was not so good.

I currently use UDF extensively as my cross-platform solution. No much problems to report.
Maybe there were still some bugs some years back that got correct between your experimenting and my usage ?

No journaling would be supported either, as that feature came after UDF 2.01.

I don't follow your thoughs ?! UDF is a log-structured file system, the whole filesystem is the journal.
The only thing added by post-2.01 version like 2.50 is feature to help crash/corruption recovery like higuita mentioned.

[chithanh]

I currently use UDF extensively as my cross-platform solution. No much problems to report.
Maybe there were still some bugs some years back that got correct between your experimenting and my usage ?

I don't know. It was back in the Windows Vista days (~2008-ish). But to my knowledge, the Linux UDF code did not see many changes since then, not sure about Windows.

Here is what I remember doing, maybe you can reproduce:

• Vista would not write to UDF created with mkudffs (interestingly, it worked with newfs_udf from FreeBSD geom).
• Linux had problems mounting UDF created by Vista if I chose the wrong block size during format.
• When I placed some files on UDF and removed world read permissions under Linux, then they became unreadable on Vista even as Administrator.
• Under Linux, heavy I/O on sparse files (could be triggered with rsync, bittorrent) caused kernel oopses and data was corrupted.
• Sometimes, I haven't found out why or how to trigger this issue, Vista would not see certain files that I copied to the filesystem in Linux. But something was there, trying to create a file of the same name as the existing non-visible one in the directory would lead to errors.

I don't follow your thoughs ?! UDF is a log-structured file system, the whole filesystem is the journal.
The only thing added by post-2.01 version like 2.50 is feature to help crash/corruption recovery like higuita mentioned.

Right, the crash recovery was what I meant.

[ant8]

I don't know. It was back in the Windows Vista days (~2008-ish). But to my knowledge, the Linux UDF code did not see many changes since then, not sure about Windows.

Here is what I remember doing, maybe you can reproduce:

• Vista would not write to UDF created with mkudffs (interestingly, it worked with newfs_udf from FreeBSD geom).
• Linux had problems mounting UDF created by Vista if I chose the wrong block size during format.
• When I placed some files on UDF and removed world read permissions under Linux, then they became unreadable on Vista even as Administrator.
• Under Linux, heavy I/O on sparse files (could be triggered with rsync, bittorrent) caused kernel oopses and data was corrupted.
• Sometimes, I haven't found out why or how to trigger this issue, Vista would not see certain files that I copied to the filesystem in Linux. But something was there, trying to create a file of the same name as the existing non-visible one in the directory would lead to errors.

Right, the crash recovery was what I meant.

Not sure if my experience is the same, but I have a 128GB USB flash drive formatted with UDF that holds my music collection. In Arch Linux, when I copy new albums, and make adjustments to vorbis tags, quite often I get random corruption of files. I have to constantly run a vorbis validation tool on the entire collection to find out if any files have been corrupted. There are no errors reported in log files. The USB drive is a branded Kingston 128GB Data Traveller flash drive.

[DrYak]

I don't know. It was back in the Windows Vista days (~2008-ish). But to my knowledge, the Linux UDF code did not see many changes since then, not sure about Windows.

On the linux side, the UDF code it-self isn't buggy. The recent changes concern tools around UDF to help get more out of it.

Vista would not write to UDF created with mkudffs (interestingly, it worked with newfs_udf from FreeBSD geom).

That is a well known limitation of Windows and Mac OS X and is the reason why you have tools assisting you such as JElchison's format-udf.

Windows can only correctly work if the UDF volume is in a partition listed in the driver's partition table and not at the begining of the drive.
(And conversely Mac OS X can only work if the UDF volume is at the beginning of the drive and not in a partition).
(I strongly suspect that their volume format auto-detection magic is broken. Probably due to the peculiarity of initial free space on UDF volumes. Their code breaks in different ways.)
Note that Linux is happy with both. So it would be happily formatting a whole drive, that will then NOT be useable correctly by Windows.

Also, older versions of Windows don't work with all versions of UDF. (e.g.: older Windows XPs, couldn't even write on any).
You might end up with a partition supported by mkudffs but not by the peculiar version of Windows you're running (or vice-versa).

Tools such as format-udf assist you by picking the correct options of UDF (e.g.: picking the lowest common denominator of versions that cover all the OS that you target. Usually stick to some 1.x versions), and by formating and partitioning in the most compatible way using some special features of UDF.

(For a more extended explanation: UDF feature a *ton* of free space at the beginning of the partition.
It servers 2 purposes :
- like on a ISO9660 volume and unlike FAT32, it means you can fit the whole boot-loader in the free space at the beginning, instead of relying on extra 1.5 stage.
- Also, it has so much space that you can fit a ISO9660 partition data in there, enabling you to make a hybrid ISO+UDF CD/DVD with both an iso9660 and a UDF structure, both pointing to file data on the same blocks (similar to how you can do an EXT4-to-BTRFS inplace conversion)
All tools like format-udf use the free space instead to fit a partition table, telling the system that there's an UDF volume beginning at sector 0.
- so there is a UDF volume beginning right at the edge of the disk, at sector 0, as expected by Mac OS X
- and there is *also* a partition table at sector 0 telling where to find the UDF volume (which incidentally also happens to be at sector 0, thanks to UDF's extra free space)
exactly as expected by Windows.
--
Note that this might cause some partitioning software to panic because partition table and partitions themselves are overlapping, but in practice it's not a problem, because UDF feature tons of free space at the beginning on purpose)

Linux had problems mounting UDF created by Vista if I chose the wrong block size during format.

This is actually the correct behaviour :
- Linux is able to detect the block size of the block device (e.g.: 512B blocks like normal old harddrives and flashdrives, or bigger (1024 to 4096) like modern native large drives or like optical media). So it will use that
- the blocksize in the header is only used for applications that can't automatically deduce the size of the blocks.
- Windows actually accepted to put some wrong shit in the blocksize (unlike FAT where it is handled automatically and correctly) and then proceeded to store data according to this incorrect number).
- Put the volume under any standard compliant OS (not only Linux) and the bogus Vista volume won't work.

That's why mkudffs and format-udf handle the block size for you (and Windows does it to, but only on FAT volumes).

When I placed some files on UDF and removed world read permissions under Linux, then they became unreadable on Vista even as Administrator.

That is actually again the correct behaviour :
- world read persmissions is control for "anybody" being able to read the files, not just the user or group owning the file.
- UDF features an extension enabling unix-style access rights, so in addition of the generic read access, you can specify a UID and a GID owning the file and special different rights for them.
- I'm not sure if plain vanilla Windows (without Unix Extensions for Winows, or wihtout Windows Services for Linux) supports this unix-style access rights extension, or if they only support ACL-style extension (just a list of arbitrary rules) (or maybe doesn't even support any extension).
- regardless of the format of access rights supported by Windows. Even if it supported Unix-style rights, "Administrator" is not the "UID_on_the_file", nor is member of the group "GID_on_the_file" and thus could not inherit their access rights. So files are correctly unreadable for "Adminstrator" on windows.
- on the other hand, as Windows' "Administrator" user is an admin, you should be able to right-click the file, go to the extended access rights/ACL view and strart tweaking the rights to get the file readable. (adain: as long as Windows supports some kinds of write management)

When the point is to exchange files accross different OS, which have entirely different concepts of access rights, and even user identities, YOU SHOULD NOT play around too much with chmods.
(Or at least you should use something that can store them as readable text stored in the XATTRs, and then some form of idmapper that can translate them into local rules. Something like fake_root or rsync's fakeroot mode And also try to match the textual names with local UIDs, like NFS' idmapper is doing).

Under Linux, heavy I/O on sparse files (could be triggered with rsync, bittorrent) caused kernel oopses and data was corrupted.

Saddly, on CoW / Log-structured file systems, sparse files and/or heavy-written-to files aren't trivial to handle.
Data isn't overwritten at the desired position (as with ext4 partitions), but a new entry is added on the log-structure/a new copy is generated, and the extent tree is updated (and at some point in time, the unused extents are garbage collected).

Means that such a torrent file will not progressively get filled on disk, instead the file will by a complex maze of extents pointing to the next chunk in download-chronological order, instead of the topological order.
To read the correct piece of data in such a file, the filesystem driver needs to traverse the whole maze until it finds the correct extent.
Probably older version of the driver were more buggy and might crash on a too big maze (also some older verisons of btrfs have similar bugs) (ZFS eats RAM and CPU for breakfast for this reason among others)

Thus :
- UDF (and other cow/log structure) isn't the best appropriate filesystem for such heavy work.
- USB flash drivers aren't either the best medium (constant I/O on a medium that doesn't support re-writes puts a stress on the translation layer, and might also cause a bit early failure if the the TL isn't that good).
- file should be marker as noCow on system that support it (e.g.: btrfs, zfs)
- file should be defragged after creation (standard practice for btrfs in post-download scripts)
- auto-defragging should be turned on (btrfs, zfs)
I don't no how the later are done with UDF: I haven't researched it, because I don't run into it as I only use UDF for data exchange and store IO intensive work (e.g.: a VM harddrive, SQL DB) on a BTRFS partition with the correct noCow attribute set on relevant files.

And more so:
Use a recent kernel with a debugger filesystem driver that can actually survive such complex mazes.

Right, the crash recovery was what I meant.

Ah okay, then.
But note that the log features added in 2.50 are rather minor. Due to its log-structured nature, UDF is already quite resilient.