Proposed Reflink Support Would Provide Big Space Savings For Wine

[pal666]

What they say is that zfs is fast enough that the bottleneck is the disk itself, and adding this syscall merely helps with deduplicate blocks.

no, they said they can only do implementation which will be not much faster than no implementation, i.e. not much faster than read of source file followed by write of destination file. they could skip only memory copy, but before doing memory copy you have to read from storage to memory. on serious filesystems reflink avoids reading and writing file blocks, it just adjust pointers.

They are not saying that it is undoable, since CoW filesystem, by its nature, split files into blocks so that on write, only the blocks that are updated and the meta blocks need to be added to the disk as new blocks.

it was already said this has nothing to do with cow. besides btrfs target users of this syscall are xfs and nfs

there’s nothing by design that ZFS cannot do.

lol. you should organize zfs sect. there was an lwn article from ex-zfs dev in 2009 which i posted here explaining its design deficiencies.

[binarybanana]

Another good option for shared storage is UDF. It's meant for DVD originally, but it works well for all kinds of media, including USB, HDD and SSD. It supports links and UNIX permissions among other stuff. Technically even something like (frozen) snapshots should are possible. Full RW support is available on almost any notable OS. Probably the most widely supported FS by far. It's surprising how little it's used.

If you're really hardcore and need to access a LUKS/LVM/bcache/btrfs volume or something similarly convoluted on Windows a small Linux VM that mounts and exports that over NFS (or SMB) also works well. Performance with VBox and qemu sucks badly though, but VMware easily pushes hundreds of MB/s over the virtual network.

[intelfx]

What they say is that zfs is fast enough that the bottleneck is the disk itself, and adding this syscall merely helps with deduplicate blocks.

They are not saying that it is undoable, since CoW filesystem, by its nature, split files into blocks so that on write, only the blocks that are updated and the meta blocks need to be added to the disk as new blocks.

They could easily support reflink by simply copy over references to the old blocks.

BTRFS also uses the same way since it is also a CoW filesystem, and if BTRFS can do it, there’s nothing by design that ZFS cannot do.

That's a very simplistic view of the matter. There's enough differences between btrfs and zfs despite both being based on the copy-on-write principle.

And no, what they say is exactly what they say — that zfs does not (yet) support reflinks, not that zfs is "fast enough" or anything else.

[landeel]

I just mounted Windows NTFS/FAT partition under Linux and used it that way. Surprisingly this worked well with Witcher 3 on Windows.

I'm currently using NTFS, but it has a few problems:
1) Bad performance if you use the NTFS-3g fuse driver;
2) Unstability, data corruption if you use the upcomming "NTFS3" driver (from my experience, I hope it gets better);
3) Symlinks made with NTFS-3g and NTFS3 are not compatible with each other, and not compatible with Windows. Makes a mess! ;
4) Steam/Proton compatdata (wine prefixes) doesn't seem to work correctly on NTFS.

[oiaohm]

So how would this actually work? When I create a new prefix, where does it get the original files to ref-link? ~/.wine is the default Wine prefix, but not necessarily one created with the same Wine version used to create the new prefix... I just don't understand how this could work without a Wine manager that knows which profile are using same Wine version and which are not. Will they be ref-linked from the system-installed package? I guess that should be acceptable for those that only have 1 version installed at a time.

Yes this would be reflink to the system package/packages. /opt/wine-devel/lib64/wine/x86_64-windows/ and /opt/wine-devel/lib/wine/x86_64-windows/ as possible paths.

https://www.winehq.org/pipermail/wine-devel/2021-July/191267.html

The vast majority of these files are byte-for-byte identical to Wine's central DLL copies.

Do note when you run a different version of wine over a wine prefix and you see updating wine prefix most of these central dll are being replaced in the prefix. There is are reasons why its not recommend to use multi wine versions in the same wine prefix. Problem 1 is the dlls in the wine prefix are version particular and at times you will run into the event where the application loads the system wide and the system32 in wineprefix at the same time and if they don't match problem. Wine is fairly good at updating wineprefix directories very bad at downgrading wineprefix directories because libraries from newer versions of wine can be left in place with a old wine version down grade then do something the old wine version totally hates. Next is the differences in registry entries between wine versions its the same kind of problem is this unknown to old wine registry entry something a application added or is this something a new version of wine added this can become a very hard question to answer

Of course if your location where wine is system wide installed is on a different partition to your home directory this relink solution in it current form is not going to work.

The explosion in wine prefix size is directly linked to the move from wine pe/elf to pe Yes in the pe/elf day the dll files in the wine prefix were a lot smaller due to being fakes with as small as code as possible just to redirect to the pe/elf binary bad new copy protection stuff hated finding elf headers and dll stacking. What use to be under 5 meg per wine prefix elf/pe with fake dlls has exploded to 150-300 megs in the PE dll time frame yes reflinking brings this back to 1 meg. The old fakedlls were not exactly the most disc effective option.

I see this as first step. I do wonder if support of a wine user to reflink against on a users home directory when that home directory is a different partition may be a good thing but I see this as a future problem. Please note you really don't want to be reflinking masters from something in the users home directory that could have been damaged by a install of a program when attempting to diagnose issues. With the way reflink works you get a error wine a reflink cannot be done so it is possible to have multi master copies with 1 master copy per partition and systematically attempt reflink after reflink until one works and if non works drop back to copy.

https://www.ctrl.blog/entry/file-cloning.html

Please note this reflink support long term is not only need to solve the wineprefix size explosion caused by going PE. It is also need to support applications expecting refs2 and to use "DUPLICATE_EXTENTS_TO_FILE" feature under windows. Yes you need applications to know if a file has managed to be de-duplicated or not. Yes some applications will alter their operations when they find that X location will not de-duplicate straight away.

So reflink support by some means over time will come a more required feature.

[geearf]

So how would this actually work? When I create a new prefix, where does it get the original files to ref-link? ~/.wine is the default Wine prefix, but not necessarily one created with the same Wine version used to create the new prefix... I just don't understand how this could work without a Wine manager that knows which profile are using same Wine version and which are not. Will they be ref-linked from the system-installed package? I guess that should be acceptable for those that only have 1 version installed at a time.

[NobodyXu]

Merged dir lookup is cached, there shouldn't be more that 3-4 layers. Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown

I can’t find any number regarding overlayfs on ext4 vs native or btrfs vs ext4, but if you are using overlayfs, I would say the performance of overlayfs can be as bad as btrfs, or theoretically, even worse.

Reading the file is indeed fast once it is opened, but if you want to read and write, it gets complicated.

Overlayfs operates on file, which means that if a file is modified, the whole file is copied into the up dir, then modified in place, instead of only copying the blocked being modified and the meta blocks, which essentially means either overlayfs uses copy_file_range to optimise the copying (which I remember it does support), or it ends up just using as much space as not using it, if the modifications are small enough that most of the blocks can be shared under CoW filesystem.

IMHO using copy_file_range is the best options.

Doesn’t force you to use overlayfs (which still takes time when opening up new, uncached file) or force you to use CoW filesystem.

Instead, anything supports copy_file_range is usable, like xfs.

[discordian]

Using overlayfs makes file opening slower due to the fact that it needs to lookup in different layers, but the read/write performance is just as good as native.

So if your game repeatedly open new files, then using overlayfs might see significant slowdown.

Merged dir lookup is cached, there shouldn't be more that 3-4 layers. Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown

[NobodyXu]

that's not what they said. they said zfs is incapable of providing reflink support, i.e. they could support syscall, but it will be as slow as without it

This system call merely removes the copies from user and kernel spaces. In most situations that's not going to be substantially faster than your disks or controllers can perform the IO.

What they say is that zfs is fast enough that the bottleneck is the disk itself, and adding this syscall merely helps with deduplicate blocks.

They are not saying that it is undoable, since CoW filesystem, by its nature, split files into blocks so that on write, only the blocks that are updated and the meta blocks need to be added to the disk as new blocks.

They could easily support reflink by simply copy over references to the old blocks.

BTRFS also uses the same way since it is also a CoW filesystem, and if BTRFS can do it, there’s nothing by design that ZFS cannot do.

[NobodyXu]

One of my pet beefs with wine. Still making the feature dependent on filesystem features is ackward.

I would prefer the docker approach of using overlays. Would be quite handy in case of updates to the underlying base filesystem.

But having an AMD GPU, that's of course all theoretical.

Using overlayfs makes file opening slower due to the fact that it needs to lookup in different layers, but the read/write performance is just as good as native.

So if your game repeatedly open new files, then using overlayfs might see significant slowdown.