Proposed Reflink Support Would Provide Big Space Savings For Wine

[oiaohm]

I always thought Wine needed some kind of virtual file system or overlay. It could work like reflinks but also be used for stuff like winetricks where you would have to install painful stuff like dotnet (still broken) only once and to reuse it you just add an override on top of another bottle. It would need to go beyond the filesystem probably like it touches the registry and who knows what else.

This turns out to be a lot harder. Please be aware that people changing between a lot of different questionable software end up having to complete reinstall windows as windows users. The problem file system and registry for most programs. The problem comes what is truly global set and what per user. Yes people do find particular things like some programs if you have multi users on windows and use a different user and that user does not have access to the other users home/profile directory the software dies. Because in system wide setting the application put user particular directories or user particular preferences. Yes this could be in the programs folder in a configuration file or in registry.

Yes this behaviour can change with software updates. So reuse is a lot harder. Yes being able to install a bottle/prefix and duplicate is a feature wine supports. relink copy a wine prefix I have done before xfs. This starts showing you how often programs do horrible things because files installed by something like dotnet you see programs modify global setting(as in files installed by the likes of dotnet' to suit their needs. Yes this global modification explains why if windows users installs a mixture of random applications they roll snake eyes from time to time. Yes the wine install 1 application per prefix is a diagnostic path to work out there is software conflict.

In a ideal software world you should be able to install all windows software in one wine prefix that don't require overrides. The reality is this does not work even under windows. Wineprefix is a work around to wine limitations and a windows limitation.

AHOY --reflink=always and --reflink=auto of the cp command are very useful when on a relink supporting file system to copy wineprefix/bottle with to keep sizes down to a point but as wine prefixes update you will notice they will also split from each other and you will have to run a deduplication pass to merge them back to each other.

Other thing to consider a runtime virtual file system does come with a overhead cost. relink route with a de-duplication tool is very light in overhead because you are working with a feature the file system supports.

Yes wine virtual file system for faking up case name insensitivity is very expensive and faking up something like reflinks is even more expensive as it would result in scanning more and more directories and having to do copy operations. AHOY there are just cases like case-folding and keeping wine prefix de-duplicated as possible you really do need for best performance file system support as native file systems can do this in synced structures without horrible overhead. The problem here is between the time you last scanned the file system and now how do you know if you are still current. File system global state is the problem to replicate as a layer on top. Global state as the file system itself is really simple to implement.

Do note even if wine did implement own overlay/reflinks in userspace you would still have the problem of needing de-duplication tool to merge prefixes back to only have 1 copy of a file when a file was updated in 2 or more prefixes to the same file by different applications.

Yes using btrfs or xfs reflink feature has manually been usable with wine for quite some time to keep prefix size down. Supporting auto reflinking when file system supports it will be beneficial. Even so after auto reflinking you still need a de-duplicate tool to scan over wine prefixes to find files that have been duplicated that need to be merged back for absolute ideal disc usage.

https://www.ctrl.blog/entry/file-cloning.html

The annoying part about ZFS here is oracles ZFS does support reflink yet openzfs freebsd zfs and zfs for Linux don't. Yes ZFS on disc format has the structures to support reflinking.

EXT4 problem is it does not have the disc structures to effectively support reflinking a developer had a look at it again in 2014 and it was found you add reflink support to ext4 you have to add a wrapper layer to patch over the disc structures limitations that would result in lower performance. I am not talking small percentage drop here I am talking write operations being like 10-15% slower.

There is no ideal file system for wine under Linux. Case-folding limited file system support and reflink support limitations are both problems. Ideal file system for wine would support both.

[pal666]

According to https://github.com/openzfs/zfs/discussions/4237, they just don’t bother to add it as it does not improve performance significantly.

that's not what they said. they said zfs is incapable of providing reflink support, i.e. they could support syscall, but it will be as slow as without it

[pal666]

There are many people who prefer old style partitioning with /home and / on mounted on different partitions. Easier to do clean reinstall that way. Reflinks don't work across different filesystems.

btrfs supports superior way of partitioning: /home and / on different subvolumes of same partition, which will not require you to ask your crystal ball how to split available space between partitions. it will support same easy clean reinstalls and it will support reflinks(if you will not mount subvolume separately, i.e. just make subvolume at /home but do not mount it)

[oiaohm]

that's not what they said. they said zfs is incapable of providing reflink support, i.e. they could support syscall, but it will be as slow as without it

https://www.ctrl.blog/entry/file-cloning.html

That at straight up lie with the incapable bit. Zfs by design is capable of providing reflink. Oracle ZFS has the feature. The data structs of ZFS include the feature. The open source implementations just don't implement it. Having the function implemented we can see with Oracle ZFS that most of time it no faster and no slower than without reflink. But there are select usage cases where having reflink support ends up faster with the Oracle ZFS.

[pal666]

That at straight up lie with the incapable bit. Zfs by design is capable of providing reflink. Oracle ZFS has the feature. The data structs of ZFS include the feature. The open source implementations just don't implement it. Having the function implemented we can see with Oracle ZFS that most of time it no faster and no slower than without reflink. But there are select usage cases where having reflink support ends up faster with the Oracle ZFS.

i don't really care how much they have to implement to make it work, the fact is they didn't do it and they said they can't. i.e. their words can't be used as proof of "zfs supports reflink" which was attempted in comment to which i replied

[oiaohm]

i don't really care how much they have to implement to make it work, the fact is they didn't do it and they said they can't. i.e. their words can't be used as proof of "zfs supports reflink" which was attempted in comment to which i replied

The problem here is the oracle version of ZFS does support reflink and does use the same disc format. Yes oracle version of ZFS that have at times made work with Linux did in fact have the syscall working.

The reality its just a openzfs and zfs for Linux has chosen not to.
https://www.ctrl.blog/entry/file-cloning.html It is noted here that orcale zfs supports it.

Use cases why exposing reflink is important is the cases where you need to know if there is one or more copies of a particular data on storage. Yes de-duplicating is good to a point but when you get to the point that something does in fact need 3 copies because it critical to deal with bit flip problems not so good.

https://lwn.net/Articles/332802/ Where does Linux kernel reflink implementation come from none other than Oracle.

https://docs.oracle.com/cd/E88353_01/html/E37843/reflink-3c.html

Yes the Linux implementation of reflink design is directly based on the Solaris design of reflink. Yes the Solaris reflink functions are only for use with ZFS at the start under Solaris.

The reality here ZFS as a file system support reflink and did from day one. The third party drivers implementing Zfs such as openzfs and zfs for Linux have chosen not to support this. Ok under BSD platforms that don't have a reflink syscall this does make sense. But on Linux that has a reflink syscall based on the solaris reflink syscall.... this is a different matter right.

[NobodyXu]

One of my pet beefs with wine. Still making the feature dependent on filesystem features is ackward.

I would prefer the docker approach of using overlays. Would be quite handy in case of updates to the underlying base filesystem.

But having an AMD GPU, that's of course all theoretical.

Using overlayfs makes file opening slower due to the fact that it needs to lookup in different layers, but the read/write performance is just as good as native.

So if your game repeatedly open new files, then using overlayfs might see significant slowdown.

[NobodyXu]

that's not what they said. they said zfs is incapable of providing reflink support, i.e. they could support syscall, but it will be as slow as without it

This system call merely removes the copies from user and kernel spaces. In most situations that's not going to be substantially faster than your disks or controllers can perform the IO.

What they say is that zfs is fast enough that the bottleneck is the disk itself, and adding this syscall merely helps with deduplicate blocks.

They are not saying that it is undoable, since CoW filesystem, by its nature, split files into blocks so that on write, only the blocks that are updated and the meta blocks need to be added to the disk as new blocks.

They could easily support reflink by simply copy over references to the old blocks.

BTRFS also uses the same way since it is also a CoW filesystem, and if BTRFS can do it, there’s nothing by design that ZFS cannot do.

[discordian]

Using overlayfs makes file opening slower due to the fact that it needs to lookup in different layers, but the read/write performance is just as good as native.

So if your game repeatedly open new files, then using overlayfs might see significant slowdown.

Merged dir lookup is cached, there shouldn't be more that 3-4 layers. Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown

[NobodyXu]

Merged dir lookup is cached, there shouldn't be more that 3-4 layers. Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown

I can’t find any number regarding overlayfs on ext4 vs native or btrfs vs ext4, but if you are using overlayfs, I would say the performance of overlayfs can be as bad as btrfs, or theoretically, even worse.

Reading the file is indeed fast once it is opened, but if you want to read and write, it gets complicated.

Overlayfs operates on file, which means that if a file is modified, the whole file is copied into the up dir, then modified in place, instead of only copying the blocked being modified and the meta blocks, which essentially means either overlayfs uses copy_file_range to optimise the copying (which I remember it does support), or it ends up just using as much space as not using it, if the modifications are small enough that most of the blocks can be shared under CoW filesystem.

IMHO using copy_file_range is the best options.

Doesn’t force you to use overlayfs (which still takes time when opening up new, uncached file) or force you to use CoW filesystem.

Instead, anything supports copy_file_range is usable, like xfs.