Proposed Reflink Support Would Provide Big Space Savings For Wine

[Berniyh]

Many games need additional dlls. Most of my Proton prefixes have 500+ MB.

I'm not sure if it works well for Proton, at least with the way how it's currently distributed.
afaik reflinks don't work in tar balls, so they would have to change that such that the reflinks are created afterwards.
That could make it incompatible with custom proton builds like glorious eggroll, though?
But yeah, the distribution size of proton versions IMO would be a bigger argument actually than the space savings on the disk.

Also, there are very small games (1-100MB). The prefix shouldn't take more space than the game itself.
So, 10 games installed = 3-5 GB of bloat.
It does make a huge difference for people who buy the 64 GB model of Steam Deck.

Sure, that's why I said that there are surely corner cases where this really matters.
Of course I don't know how much of a corner case the Steam deck will be. We'll see.

[geearf]

Yes this would be reflink to the system package/packages. /opt/wine-devel/lib64/wine/x86_64-windows/ and /opt/wine-devel/lib/wine/x86_64-windows/ as possible paths.

https://www.winehq.org/pipermail/wine-devel/2021-July/191267.html

Do note when you run a different version of wine over a wine prefix and you see updating wine prefix most of these central dll are being replaced in the prefix. There is are reasons why its not recommend to use multi wine versions in the same wine prefix. Problem 1 is the dlls in the wine prefix are version particular and at times you will run into the event where the application loads the system wide and the system32 in wineprefix at the same time and if they don't match problem. Wine is fairly good at updating wineprefix directories very bad at downgrading wineprefix directories because libraries from newer versions of wine can be left in place with a old wine version down grade then do something the old wine version totally hates. Next is the differences in registry entries between wine versions its the same kind of problem is this unknown to old wine registry entry something a application added or is this something a new version of wine added this can become a very hard question to answer

Of course if your location where wine is system wide installed is on a different partition to your home directory this relink solution in it current form is not going to work.

The explosion in wine prefix size is directly linked to the move from wine pe/elf to pe Yes in the pe/elf day the dll files in the wine prefix were a lot smaller due to being fakes with as small as code as possible just to redirect to the pe/elf binary bad new copy protection stuff hated finding elf headers and dll stacking. What use to be under 5 meg per wine prefix elf/pe with fake dlls has exploded to 150-300 megs in the PE dll time frame yes reflinking brings this back to 1 meg. The old fakedlls were not exactly the most disc effective option.

I see this as first step. I do wonder if support of a wine user to reflink against on a users home directory when that home directory is a different partition may be a good thing but I see this as a future problem. Please note you really don't want to be reflinking masters from something in the users home directory that could have been damaged by a install of a program when attempting to diagnose issues. With the way reflink works you get a error wine a reflink cannot be done so it is possible to have multi master copies with 1 master copy per partition and systematically attempt reflink after reflink until one works and if non works drop back to copy.

https://www.ctrl.blog/entry/file-cloning.html

Please note this reflink support long term is not only need to solve the wineprefix size explosion caused by going PE. It is also need to support applications expecting refs2 and to use "DUPLICATE_EXTENTS_TO_FILE" feature under windows. Yes you need applications to know if a file has managed to be de-duplicated or not. Yes some applications will alter their operations when they find that X location will not de-duplicate straight away.

So reflink support by some means over time will come a more required feature.

Thanks for this full explanation, much appreciated!
But yup you're right, Wine is installed on /, games are either on on the same SSD but different partition, or on an HDD, that's not useful :'/
I guess it'd be good to have auto deduplication of the FS, one that doesn't create too much fragmentation I guess..

[oiaohm]

Well yes, if you want to get totally unnecessarily pedantic, then "to have reflinks" is just a colloquial shorthand for "to have reflink copies", or even more precisely "to support creating reflinks, or otherwise implement reflink semantics, via Linux-specific copy acceleration syscalls".

This is wrong. reflinks was is a short hand for reference links. This is a name to explain a file system feature. You have 3 types of reference linking. Hard linking, symbolic linking and reflinks. Yes reflinks place holder name.

https://en.wikipedia.org/wiki/NTFS_links Like windows XP supports symbolic links in the ntfs driver but does not expose them to user-space. This does not mean that Windows XP did not support symbolic links just because it was not a user space feature.

ZFS [as present on Linux] may have block deduplication features for all it wants, but in colloquial sense it doesn't "support reflinks" unless this functionality is exposed via said syscalls.

ZFS on Linux de-duplication feature is running on reflinks. If I make ZFS file system on Oracle Solaris and create use a reflink ZoL will hand it correctly. If someone created a third party tool to put a reflink on a ZFS volume again ZoL would handle it correctly. The de-duplication process code when it detects a matched up file at times will run a reflink command with ZoL as well.

So the functional is there. Only thing missing is the developers of ZoL don't see it and they don't export the syscall. Sorry the Linux copy acceleration syscalls are based on the Solaris ones.

The reality here is reflink is part of the core ZFS design. First file system with reflink was ZFS the second file system with reflink is xfs.

There is a completely different problem here. ZoL developers don't want to say what the problem is. Hooking up Linux kernel reflink interfaces that are under GPLv2 only to CDDL only ZFS deduplication code where the ZFS reflink code is happened to be very legally problematic. Telling users to use other features or there is no gains to the reflink feature means they don't have to say they are license stuck.

ZFS by design has reflink. If file system driver exposes that to user space that a different matter. Of course when you understand the license problem here you understand why the ZoL developers are attempting to weasel their out of having to implement reflink as userspace usable feature even that they are using reflink feature internally.

[intelfx]

This is also false. ZoL does have reflinks just not exposed as the Linux syscall.

Well yes, if you want to get totally unnecessarily pedantic, then "to have reflinks" is just a colloquial shorthand for "to have reflink copies", or even more precisely "to support creating reflinks, or otherwise implement reflink semantics, via Linux-specific copy acceleration syscalls".

ZFS [as present on Linux] may have block deduplication features for all it wants, but in colloquial sense it doesn't "support reflinks" unless this functionality is exposed via said syscalls.

The reality here is reflink has nothing todo with being a CoW based file system instead everything to with block level de-duplication. Yes a block level de-duplication is absolutely useful to a CoW based filesystem but that feature does not mean you have to.

You may have noticed that I never claimed otherwise.

[oiaohm]

OK, good. ZoL still doesn’t have them though.

This is also false. ZoL does have reflinks just not exposed as the Linux syscall. ZoL is a port of the Solaris ZFS to Linux. The de-duplication code in zfs uses sun designed reflinks yes that includes ZoL and openzfs. Where reflinks syscall is meant to hook to by Solaris for Sun and Orcale is the "Block level deduplication" bit of ZFS not the snapshot-ting system.

The reality is ZoL developers don't understand the problem like they suggest using clone instead and other things that are strictly not part of how ZFS is meant to have reflinks implemented.

The reality is openzfs and ZoL both do have reflinks in openzfs case of being used on BSD you can excuse not export reflinks because there is not a reflinks syscall on most BSD solutions. Now on Linux where you do have reflinks syscall and you do have a block level de-duplication system you are able to export reflinks.

Remember xfs is not a traditional CoW file system instead XFS a non CoW file system with block level de-duplication under it this is how it also implements reflinks.

The reality here is reflink has nothing todo with being a CoW based file system instead everything to with block level de-duplication. Yes a block level de-duplication is absolutely useful to a CoW based filesystem but that feature does not mean you have to.

Like if ext4 was sitting on VDO what is a redhat de-duplication block device it would be possible to implement reflinks with ext4 without major alteration to ext4. Think of it this way reflink syscall is the means to say I need block ID for this data so that this data if modified will CoW for the file system to use . Remember VDO is not mainline Linux.

[intelfx]

Orcale ZFS does support reflinks and SUN version of ZFS for Solaris also supported reflinks. This is before openzfs and zfs for Linux and zfs driver in freebsd existed. This is more the other implementations of ZFS have not implemented a feature the on disc format of ZFS in fact supports. Linux kernel support for reflinks is part based on the Solaris stuff.

OK, good. ZoL still doesn’t have them though.

[oiaohm]

And no, what they say is exactly what they say — that zfs does not (yet) support reflinks, not that zfs is "fast enough" or anything else.

Orcale ZFS does support reflinks and SUN version of ZFS for Solaris also supported reflinks. This is before openzfs and zfs for Linux and zfs driver in freebsd existed. This is more the other implementations of ZFS have not implemented a feature the on disc format of ZFS in fact supports. Linux kernel support for reflinks is part based on the Solaris stuff.

[oiaohm]

Another good option for shared storage is UDF. It's meant for DVD originally, but it works well for all kinds of media, including USB, HDD and SSD. It supports links and UNIX permissions among other stuff. Technically even something like (frozen) snapshots should are possible. Full RW support is available on almost any notable OS. Probably the most widely supported FS by far. It's surprising how little it's used.

https://en.wikipedia.org/wiki/Universal_Disk_Format#Compatibility

Not really there is a nice little catch here. Yes Linux can read 2.50 and 2.60 UDF but cannot write it. Windows can auto update UDF 2.0x to 2.50/2.60. This leads to the fun you were able to read write 2.0x in Linux you boot into windows copy a file and now its read only when you return to Linux. This is another case driver support is problem. Please note mac os also says stuff you if windows upgraded UDF to 2.60

Using a file server like SMB/NFS is way less of odd ball stuff happening when sharing files between OS..

[oiaohm]

4) Steam/Proton compatdata (wine prefixes) doesn't seem to work correctly on NTFS.

https://wiki.winehq.org/FAQ#Does_Wine_run_on_all_Unix_filesystems.3F

That one is answered in the wine FAQ. NTFS-3g is missing a feature lot steam hosted applications/games expect to work. This does not matter if it steam proton or the game installed in wine directly with some of these feature either. Yes using ext4 as a loop back file on a NTFS to run wine games can stupidly work out better than running straight on ntfs-3g.

[pal666]

Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown

btrfs is faster than ext4 when you measure important things