Proposed Reflink Support Would Provide Big Space Savings For Wine

[intelfx]

I'm considering using a BTRFS partition to share my data and game library between Linux and Windows.
Does anyone know how stable and secure WinBtrfs is? Risks of data loss?

It used to corrupt filesystems on trivial operations in the past and there are some troubling open bugreports too. I have yet to try a recent build on one of my systems — the only Windows installations I have right now are in VMs.

[ElectricPrism]

If this makes debugging WINE prefixes easier by having a less complex file-tree I will be very happy.

Lately I have been using using App/Prefixes/Main where Main directory is a BTRFS sub-volume and then snapshotting to say Main_2021-07-25_12-00 prior to experimental changes to the prefix before I bork the prefix forever.

Prior to this I simply had to keep all prefix changes documented in a INSTALL.md and then literally throw out the whole prefix every time something went wrong.

If it reduces overlay clutter it will make it __ MUCH ___ easier to __ DIFF __ Prefix_A and Prefix_B before and after a change occurs which will help to understand what changes are prefix-breaking and which aren't.

I'm so happy to see this patch coming, I've longed for these kinds of improvements which are already a thing in Darling for YEARS. I can't wait to try it out.

[cl333r]

If this makes debugging WINE prefixes easier by having a less complex file-tree I will be very happy.

Lately I have been using using App/Prefixes/Main where Main directory is a BTRFS sub-volume and then snapshotting to say Main_2021-07-25_12-00 prior to experimental changes to the prefix before I bork the prefix forever.

Prior to this I simply had to keep all prefix changes documented in a INSTALL.md and then literally throw out the whole prefix every time something went wrong.

If it reduces overlay clutter it will make it __ MUCH ___ easier to __ DIFF __ Prefix_A and Prefix_B before and after a change occurs which will help to understand what changes are prefix-breaking and which aren't.

I'm so happy to see this patch coming, I've longed for these kinds of improvements which are already a thing in Darling for YEARS. I can't wait to try it out.

Afaik you the end user will see this as: the list of files and their sizes will look exactly the same, except that now it "magically" uses less space that's because i.e. a 5MiB file under the hood will really only use like a few KiB or less because the data it contains is shared with the original file, which to me as a file browser creator poses yet another non-trivial problem of detecting how much space this or that really takes up.

On a related note:
copy_file_range() [1]: gives filesystems an opportunity to implement "copy acceleration" techniques, such as the use of reflinks (i.e., two or more inodes that share pointers to the same copy- on-write disk blocks) or server-side-copy (in the case of NFS).

For example copying a 2GiB movie with this call to the same partition happens in a blink of an eye using almost no space on my HDD, while a normal copy operation would take a minute and use 2GiB.

[1] https://man7.org/linux/man-pages/man...e_range.2.html

[Developer12]

AFAIK the in-band deduplication requires lots of resources — lots of memory at least, though I assume it might be CPU-heavy too. It is definitely not something a usual user would make use of.

The general rule of thumb for active block deduplication used to be 1GB per TB, but I'm not sure that's still relevant. Supposedly ZoL/OpenZFS have put a lot more work into reigning this in than the BSD implementation they recently replaced, and the way older Oracle version. (part of the work to wean ZFS off of virtual memory use or something) ZFS is also supposed to give memory back if needed, though the rate of reclaim is an active area of work.

One thing you may have missed in my previous post though is that ZFS does _multiple_ kinds of deduplication. There's block deduplication which is toggleable, but ZFS (being CoW by nature) will _always_ copy files by reference until they've been written to. I'm reasonably sure that when this happens it only copies the altered blocks. This passive always-on deduplication is perfectly relevant to WINE's copying of core files to build new prefixes, especially if they're never written.

There's also compression worth mentioning, which has to be turned on and does have CPU overhead, but with more recent, faster compression being available this is probably not a huge deal.

[Hi-Angel]

One thing you may have missed in my previous post though is that ZFS does _multiple_ kinds of deduplication. There's block deduplication which is toggleable, but ZFS (being CoW by nature) will _always_ copy files by reference until they've been written to. I'm reasonably sure that when this happens it only copies the altered blocks. This passive always-on deduplication is perfectly relevant to WINE's copying of core files to build new prefixes, especially if they're never written.

What people refer to as "CoW filesystem" is orthogonal to copying files by references aka doing reflinks.

"CoW filesystem" usually refers to a filesystem that upon changing a file will not actually overwrite its blocks, but instead it would create ones with the new content elsewhere. Thus you have both new and old blocks, and can rollback to the older version of data.

reflinks are indeed related to CoW in a sense that they employ CoW behavior where file blocks are only copied on write. However: α) this does not require a CoW filesystem, for example XFS supports it; and β) this does not have to be supported by a CoW filesystem, for example ZFS does not support it.

It would seem to me like once you have a CoW FS, adding reflinks support there should be absolutely trivial. However, if you look at the ZFS bugreport (provided in the comment you quoted), you might find it was reported in 2011 year. I admittedly didn't delve into it, but judging by the age and the amount of comments, there are apparently technical hurdles. Which even further highlights the point, that CoW and reflinks are orthogonal features.

[Up123]

Well, yes and no.

Of course these space savings are nice, especially if the space was unnecessarily occupied.
However, most games bring GBs to the disk, in some cases even tens or hundreds of GBs, so saving 300MB won't be a big deal.
Oh and HDDs and even SDDs are pretty cheap these days, too.
In some corner cases it might really help, though.

not all games are that big, I'm planning to play smaller games on it e.g. Slay the spire is tiny

[Developer12]

What people refer to as "CoW filesystem" is orthogonal to copying files by references aka doing reflinks.

As far as I know, ZFS does both. The CoW theme is pervasive in it's design. The previous issue may still be open because people are asking for a very specific BTRFS interface to be implemented.

[Hi-Angel]

As far as I know, ZFS does both. The CoW theme is pervasive in it's design. The previous issue may still be open because people are asking for a very specific BTRFS interface to be implemented.

Okay, if you say it should work, then please elaborate what system call userspace is required to use for that, if it's not the one that BTRFS/XFS are using? Reason I'm asking is because if you run `strace cp $(which bash) 2` you will find out it does copy by repeatedly reading the source file to a buffer, then writing to a destination (that's as of 8.32, in the next release it should default to reflink if available). There is no way a file system could deduplicate that unless in-band deduplication is enabled. If you say it works for some utility, then I am curious how.

[rabcor]

Yes, please.

[pal666]

I wonder why symlinking is a problem. When you do copy a file on top of a symlink with the same name, symlink will be just replaced by that file without touching the original, right?

wrong. there are many ways to copy file. if you open symlink for writing, you open base file. also symlink is visible as symlink, reflink is normal file