Announcement

Collapse
No announcement yet.

Proposed Reflink Support Would Provide Big Space Savings For Wine

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • oiaohm
    replied
    Originally posted by Berniyh View Post
    Are filesystem differences with regard to speed on a Desktop PC even relevant these days?
    After all we have very fast SSDs these days for affordable prices.

    Edit: and personally I'm even using SATA SSDs on my gaming PC for some games and can't observe any noticeable difference to the M.2 SSDs.
    On my desktop PC I'm using btrfs on LUKS-encrypted devices (which limits the throughput to about 1 GB/s on a Ryzen 5000) and still can't notice any significant performance issues.
    So... for a desktop, I think that filesystem "speed" is highly overrated. I don't think I'll ever setup a PC with a filesystem other than btrfs again.
    Most games cache into ram well with many stunts to hide their loading from storage. So games observe difference between M.2 NVME SSD and SATA SSD is insanely small. The stunts games have done to hide HDD loading once you get past a particular drive speed there is no more performance to gain for that games as it flat lines. We may get newer games designed more for NVME SSD that may show difference but this would be the games coming out with direct from NVME SSD to graphics card and the like. Of course these are likely to missing optimisations for HDD.

    Desktop PC file system performance needs change over time. The reality here is reflink if more and more application use it this can provide a performance boost to file systems that support reflink vs those that don't. Remember ext4 that does not support reflink has to perform a proper copy that results in way more IO than performing a reflink.

    File system performance needs are not exactly simple to measure. Also areas that a file system need to have performance in for desktop usage changes over time. Something to remember you may be stuck with your file system choice for a decade or more so a generally OK performance file system with broad feature set over the decade can treat you better than a file system that is ideal for today desktop performance but then ends up not ideal latter on due to lack of features.

    Yes that LUKS-encrypted that ok today is not going be workable with the future games that required direct from storage to gpu. This is the problem required features for the desktop PC keep on changing.

    Leave a comment:


  • Berniyh
    replied
    Originally posted by discordian View Post
    Merged dir lookup is cached, there shouldn't be more that 3-4 layers. Sure adding a feature adds overhead, but I think using a relink filesystem like btrfs over ext4 is the bigger slowdown
    Are filesystem differences with regard to speed on a Desktop PC even relevant these days?
    After all we have very fast SSDs these days for affordable prices.

    Edit: and personally I'm even using SATA SSDs on my gaming PC for some games and can't observe any noticeable difference to the M.2 SSDs.
    On my desktop PC I'm using btrfs on LUKS-encrypted devices (which limits the throughput to about 1 GB/s on a Ryzen 5000) and still can't notice any significant performance issues.
    So... for a desktop, I think that filesystem "speed" is highly overrated. I don't think I'll ever setup a PC with a filesystem other than btrfs again.
    Last edited by Berniyh; 28 July 2021, 12:49 PM.

    Leave a comment:


  • Berniyh
    replied
    Originally posted by landeel View Post

    Many games need additional dlls. Most of my Proton prefixes have 500+ MB.
    I'm not sure if it works well for Proton, at least with the way how it's currently distributed.
    afaik reflinks don't work in tar balls, so they would have to change that such that the reflinks are created afterwards.
    That could make it incompatible with custom proton builds like glorious eggroll, though?
    But yeah, the distribution size of proton versions IMO would be a bigger argument actually than the space savings on the disk.
    Also, there are very small games (1-100MB). The prefix shouldn't take more space than the game itself.
    So, 10 games installed = 3-5 GB of bloat.
    It does make a huge difference for people who buy the 64 GB model of Steam Deck.
    Sure, that's why I said that there are surely corner cases where this really matters.
    Of course I don't know how much of a corner case the Steam deck will be. We'll see.

    Leave a comment:


  • geearf
    replied
    Originally posted by oiaohm View Post

    Yes this would be reflink to the system package/packages. /opt/wine-devel/lib64/wine/x86_64-windows/ and /opt/wine-devel/lib/wine/x86_64-windows/ as possible paths.

    https://www.winehq.org/pipermail/win...ly/191267.html


    Do note when you run a different version of wine over a wine prefix and you see updating wine prefix most of these central dll are being replaced in the prefix. There is are reasons why its not recommend to use multi wine versions in the same wine prefix. Problem 1 is the dlls in the wine prefix are version particular and at times you will run into the event where the application loads the system wide and the system32 in wineprefix at the same time and if they don't match problem. Wine is fairly good at updating wineprefix directories very bad at downgrading wineprefix directories because libraries from newer versions of wine can be left in place with a old wine version down grade then do something the old wine version totally hates. Next is the differences in registry entries between wine versions its the same kind of problem is this unknown to old wine registry entry something a application added or is this something a new version of wine added this can become a very hard question to answer

    Of course if your location where wine is system wide installed is on a different partition to your home directory this relink solution in it current form is not going to work.

    The explosion in wine prefix size is directly linked to the move from wine pe/elf to pe Yes in the pe/elf day the dll files in the wine prefix were a lot smaller due to being fakes with as small as code as possible just to redirect to the pe/elf binary bad new copy protection stuff hated finding elf headers and dll stacking. What use to be under 5 meg per wine prefix elf/pe with fake dlls has exploded to 150-300 megs in the PE dll time frame yes reflinking brings this back to 1 meg. The old fakedlls were not exactly the most disc effective option.

    I see this as first step. I do wonder if support of a wine user to reflink against on a users home directory when that home directory is a different partition may be a good thing but I see this as a future problem. Please note you really don't want to be reflinking masters from something in the users home directory that could have been damaged by a install of a program when attempting to diagnose issues. With the way reflink works you get a error wine a reflink cannot be done so it is possible to have multi master copies with 1 master copy per partition and systematically attempt reflink after reflink until one works and if non works drop back to copy.

    https://www.ctrl.blog/entry/file-cloning.html
    Please note this reflink support long term is not only need to solve the wineprefix size explosion caused by going PE. It is also need to support applications expecting refs2 and to use "DUPLICATE_EXTENTS_TO_FILE" feature under windows. Yes you need applications to know if a file has managed to be de-duplicated or not. Yes some applications will alter their operations when they find that X location will not de-duplicate straight away.

    So reflink support by some means over time will come a more required feature.
    Thanks for this full explanation, much appreciated!
    But yup you're right, Wine is installed on /, games are either on on the same SSD but different partition, or on an HDD, that's not useful :'/
    I guess it'd be good to have auto deduplication of the FS, one that doesn't create too much fragmentation I guess..

    Leave a comment:


  • oiaohm
    replied
    Originally posted by intelfx View Post
    Well yes, if you want to get totally unnecessarily pedantic, then "to have reflinks" is just a colloquial shorthand for "to have reflink copies", or even more precisely "to support creating reflinks, or otherwise implement reflink semantics, via Linux-specific copy acceleration syscalls".
    This is wrong. reflinks was is a short hand for reference links. This is a name to explain a file system feature. You have 3 types of reference linking. Hard linking, symbolic linking and reflinks. Yes reflinks place holder name.

    https://en.wikipedia.org/wiki/NTFS_links Like windows XP supports symbolic links in the ntfs driver but does not expose them to user-space. This does not mean that Windows XP did not support symbolic links just because it was not a user space feature.

    Originally posted by intelfx View Post
    ZFS [as present on Linux] may have block deduplication features for all it wants, but in colloquial sense it doesn't "support reflinks" unless this functionality is exposed via said syscalls.
    ZFS on Linux de-duplication feature is running on reflinks. If I make ZFS file system on Oracle Solaris and create use a reflink ZoL will hand it correctly. If someone created a third party tool to put a reflink on a ZFS volume again ZoL would handle it correctly. The de-duplication process code when it detects a matched up file at times will run a reflink command with ZoL as well.

    So the functional is there. Only thing missing is the developers of ZoL don't see it and they don't export the syscall. Sorry the Linux copy acceleration syscalls are based on the Solaris ones.

    The reality here is reflink is part of the core ZFS design. First file system with reflink was ZFS the second file system with reflink is xfs.

    There is a completely different problem here. ZoL developers don't want to say what the problem is. Hooking up Linux kernel reflink interfaces that are under GPLv2 only to CDDL only ZFS deduplication code where the ZFS reflink code is happened to be very legally problematic. Telling users to use other features or there is no gains to the reflink feature means they don't have to say they are license stuck.

    ZFS by design has reflink. If file system driver exposes that to user space that a different matter. Of course when you understand the license problem here you understand why the ZoL developers are attempting to weasel their out of having to implement reflink as userspace usable feature even that they are using reflink feature internally.

    Leave a comment:


  • intelfx
    replied
    Originally posted by oiaohm View Post
    This is also false. ZoL does have reflinks just not exposed as the Linux syscall.
    Well yes, if you want to get totally unnecessarily pedantic, then "to have reflinks" is just a colloquial shorthand for "to have reflink copies", or even more precisely "to support creating reflinks, or otherwise implement reflink semantics, via Linux-specific copy acceleration syscalls".

    ZFS [as present on Linux] may have block deduplication features for all it wants, but in colloquial sense it doesn't "support reflinks" unless this functionality is exposed via said syscalls.

    Originally posted by oiaohm View Post
    The reality here is reflink has nothing todo with being a CoW based file system instead everything to with block level de-duplication. Yes a block level de-duplication is absolutely useful to a CoW based filesystem but that feature does not mean you have to.
    You may have noticed that I never claimed otherwise.
    Last edited by intelfx; 27 July 2021, 09:12 PM.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by intelfx View Post
    OK, good. ZoL still doesn’t have them though.
    This is also false. ZoL does have reflinks just not exposed as the Linux syscall. ZoL is a port of the Solaris ZFS to Linux. The de-duplication code in zfs uses sun designed reflinks yes that includes ZoL and openzfs. Where reflinks syscall is meant to hook to by Solaris for Sun and Orcale is the "Block level deduplication" bit of ZFS not the snapshot-ting system.

    The reality is ZoL developers don't understand the problem like they suggest using clone instead and other things that are strictly not part of how ZFS is meant to have reflinks implemented.

    The reality is openzfs and ZoL both do have reflinks in openzfs case of being used on BSD you can excuse not export reflinks because there is not a reflinks syscall on most BSD solutions. Now on Linux where you do have reflinks syscall and you do have a block level de-duplication system you are able to export reflinks.

    Remember xfs is not a traditional CoW file system instead XFS a non CoW file system with block level de-duplication under it this is how it also implements reflinks.

    The reality here is reflink has nothing todo with being a CoW based file system instead everything to with block level de-duplication. Yes a block level de-duplication is absolutely useful to a CoW based filesystem but that feature does not mean you have to.

    Like if ext4 was sitting on VDO what is a redhat de-duplication block device it would be possible to implement reflinks with ext4 without major alteration to ext4. Think of it this way reflink syscall is the means to say I need block ID for this data so that this data if modified will CoW for the file system to use . Remember VDO is not mainline Linux.

    Leave a comment:


  • intelfx
    replied
    Originally posted by oiaohm View Post

    Orcale ZFS does support reflinks and SUN version of ZFS for Solaris also supported reflinks. This is before openzfs and zfs for Linux and zfs driver in freebsd existed. This is more the other implementations of ZFS have not implemented a feature the on disc format of ZFS in fact supports. Linux kernel support for reflinks is part based on the Solaris stuff.
    OK, good. ZoL still doesn’t have them though.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by intelfx View Post
    And no, what they say is exactly what they say — that zfs does not (yet) support reflinks, not that zfs is "fast enough" or anything else.
    Orcale ZFS does support reflinks and SUN version of ZFS for Solaris also supported reflinks. This is before openzfs and zfs for Linux and zfs driver in freebsd existed. This is more the other implementations of ZFS have not implemented a feature the on disc format of ZFS in fact supports. Linux kernel support for reflinks is part based on the Solaris stuff.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by binarybanana View Post
    Another good option for shared storage is UDF. It's meant for DVD originally, but it works well for all kinds of media, including USB, HDD and SSD. It supports links and UNIX permissions among other stuff. Technically even something like (frozen) snapshots should are possible. Full RW support is available on almost any notable OS. Probably the most widely supported FS by far. It's surprising how little it's used.
    https://en.wikipedia.org/wiki/Univer...#Compatibility
    Not really there is a nice little catch here. Yes Linux can read 2.50 and 2.60 UDF but cannot write it. Windows can auto update UDF 2.0x to 2.50/2.60. This leads to the fun you were able to read write 2.0x in Linux you boot into windows copy a file and now its read only when you return to Linux. This is another case driver support is problem. Please note mac os also says stuff you if windows upgraded UDF to 2.60

    Using a file server like SMB/NFS is way less of odd ball stuff happening when sharing files between OS..

    Leave a comment:

Working...
X