Announcement

Collapse
No announcement yet.

The New NTFS File-System Driver Has Been Submitted For Linux 5.15

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    any benchmarks paragon vs ntfs-3g?

    or any interest in making them Michael?

    Comment


    • #32
      polarathene This patch collection seems to be a byproduct of the custom kernel the repo owner spins for the Archlinux AUR and he has also created custom code & tweaks for this project aswell (ll, lucjan).

      When you look into the README.md you can see where the patches are coming from. They are formatted in a way that you can selectively add specific features/fixes/patches of i.e. -pf, -ck, …, or archlinux kernels to a vanilla kernel.org sourcetree. A bit more information can be found on the author's gitlab here and here.

      The patches are mostly organized as diffs between the stable kernel releases (5.13, 5.14, etc.) and the develoment branches of the custom kernel projects (linux-next, ProjectC, linux-pf,… the sources mentioned in the readme). LTS versions may get updated patchsets (feature backports) via this repo for longer aswell. You can get them merged into one file to avoid clutter or separately (-sep) if you want to cherry pick for yourself.

      Since these development projects (like the ntfs patches or fixes from linux-next) do get updated several times during the lifetime of a stable kernel version or a minor release (i.e. 5.13.X) can make some of the patches obsolete, there are different versions of these patchsets (i.e. ntfs-patches-v1, -v2, -v3 and so on) aswell.

      Another reason why old versions of the patchsets are still kept there is that stuff might break or not work as expected (they're mostly using out-of-tree code or stuff that hasn't hit linux-stable, afterall) and it allows you to revert to an earlier version.

      Further you can get, say, an AIO patch containing the latest ProjectC version & dev-branch fixes without having to wait for Alfred to publish a new release as an all-in-one-patch. Be aware that lucjan structured the ProjectC patches differently than Alfred to allow for getting new features in older kernel versions (i.e. prjc-5.14 patchset in 5.13 kernels), meaning that they're incremental updates (require previous -v to be merged aswell), which is not the case for i.e. the ntfs and zstd patches.

      Alternatively, you can also get the original PDS patches (from before ProjectC) and apply them to a recent kernel. For older kernels, there are also the original BMQ patches. That's the reason why they're listed separately from each other. The original PDS patchset is what was formerly known as "undead PDS" or PDS-mq and was offered a long time for TKglitch custom kernels.

      The lru patches might be interesting for some very specific users as they are used to build realtime-capable (fully preemptible w/ latency guarantees) kernels.

      My picked the following patches: I used this repo to get some of the XFS and memory management fixes expected to land with linux-5.15 even before 5.14 stable was released, got a feature that preserves some more memory pages to avoid UI stalling when I use up all my RAM when compiling big stuff, some more block device tuning for SSDs and lower CPU utilization under heavy IO, a security tweak + other misc. stuff like more complete zstd support within the kernel. Then I added ProjectC from Alfred's gitlab (for which I may get updates from lucjan, should I want to stay on 5.13), pulled futex, fsync2 and winesync patches from TKglitch's repo for better Proton support and got the patch for generating cpu-specific optimized kernel code from graysky for a general performance boost.

      I hope this answered your questions to a satisfying degree, cheers.
      Last edited by kiffmet; 04 September 2021, 03:52 PM. Reason: typos, grammar, clarification

      Comment


      • #33
        Originally posted by piorunz View Post

        Everyone uses Windows filesystems? Sorry I am not part of your cult.
        Like it or not, the crushing majority uses Windows, which means that everyone else needs to mount Windows filesystems every now and then - whether on dual boot machines, or external hard drives etc. Some people are pragmatic and welcome the prospect of improved interoperability; and some - like you - are members of a cult that makes life more complicated for themselves for no reason at all.

        Comment


        • #34
          Originally posted by jacob View Post
          Like it or not, the crushing majority uses Windows, which means that everyone else needs to mount Windows filesystems every now and then - whether on dual boot machines, or external hard drives etc. Some people are pragmatic and welcome the prospect of improved interoperability; and some - like you - are members of a cult that makes life more complicated for themselves for no reason at all.
          There are horrible issues.
          https://superuser.com/questions/1547...ata-corruption

          There is a reason why I am asking if their will be fsck/chkdsk tool. Turns out that windows to windows with NTFS is not brilliant to the point that it will screw it up the ntfs partition when it commonly moved between versions of windows let alone moved to Linux and back. When you have something that has interoperability issues with it self and you are trying to be interoperability with it you do need tools.

          The reality if we don't have the fsck/chkdsk tools for NTFS we are better off sticking to fat32 or exfat where we do have proper repair tools for interoperability.

          Windows does not just support one file system. The means to read undamaged NTFS at speed for migrations will be useful. But as a transfer drive format not so much without the fsck/chkdsk tools.

          Comment


          • #35
            Originally posted by oiaohm View Post

            There are horrible issues.
            https://superuser.com/questions/1547...ata-corruption

            There is a reason why I am asking if their will be fsck/chkdsk tool. Turns out that windows to windows with NTFS is not brilliant to the point that it will screw it up the ntfs partition when it commonly moved between versions of windows let alone moved to Linux and back. When you have something that has interoperability issues with it self and you are trying to be interoperability with it you do need tools.

            The reality if we don't have the fsck/chkdsk tools for NTFS we are better off sticking to fat32 or exfat where we do have proper repair tools for interoperability.

            Windows does not just support one file system. The means to read undamaged NTFS at speed for migrations will be useful. But as a transfer drive format not so much without the fsck/chkdsk tools.
            Yes, Windows supports other filesystems, but for file exchange across OSes (including Windows and Mac), the options are really limited:
            • (V)FAT is a 32-bit filesystem which, among others, makes it a no-go for very large files (videos, VM images etc)
            • ExFAT had poor Linux support until relatively recently, and I'm not sure it's supported on MacOS at all at the moment (maybe read-only, which can be a blocker in many use-cases)
            • Other FSes like Ext4, F2FS require the installation of a third-party Windows driver, which may be fine if it's on a machine where you know data exchange will happen regularly, but it's no good for when a colleague says "hey mate, could you give me a copy of that VM?"
            So in practice, in many scenarios, NTFS it is. That's why I say that a new and (supposedly) much improved NTFS driver for Linux is something we should be happy about.

            Comment


            • #36
              Originally posted by jacob View Post

              Yes, Windows supports other filesystems, but for file exchange across OSes (including Windows and Mac), the options are really limited:
              • (V)FAT is a 32-bit filesystem which, among others, makes it a no-go for very large files (videos, VM images etc)
              • ExFAT had poor Linux support until relatively recently, and I'm not sure it's supported on MacOS at all at the moment (maybe read-only, which can be a blocker in many use-cases)
              • Other FSes like Ext4, F2FS require the installation of a third-party Windows driver, which may be fine if it's on a machine where you know data exchange will happen regularly, but it's no good for when a colleague says "hey mate, could you give me a copy of that VM?"
              So in practice, in many scenarios, NTFS it is. That's why I say that a new and (supposedly) much improved NTFS driver for Linux is something we should be happy about.
              ExFat macOS High Sierra that from 2017 got read/write. MacOS and Linux both have repair tools for Exfat. Mac os also has no repair tools for when NTFS screws up.

              https://github.com/relan/exfat
              exfat support on Linux has been around in fuse with read/write for quite some time with repair tools. Issue was not exactly software support here the problem was the patent issue that need to be sorted out resulting in linux distributions not shipping with exfat.

              There are a lot of cases where vfat and exfat are better options than ntfs. Yes exfat does not have the screw rate windows to windows either. Yes a file transfer external media on exfat is more likely to be successful windows to windows than using ntfs let alone the fat that this happens to support more operating systems..

              Many scenarios using ntfs is setting yourself up for hell due to lack of repair tools on Mac OS and Linux and windows to windows managing to break the NTFS in ways windows own repair tools will not fix. NTFS os to os transfer is do you fell lucky punk move where you copy a file on to a drive transport it and the thing does not work and it not due to failed media its because Windows wrote the meta data oddly or windows got upset by the way Mac OS wrote the data.

              https://www.makeuseof.com/tag/mac-fi...xternal-drive/
              The recommend transfer format from Windows to Mac OS is exfat not NTFS there are so many ways Mac OS will screw you over if you use NTFS its not funny as well.

              Linux kernel getting proper exfat support was big because it truly got Mac OS, Windows and Linux with a file system all 3 could repair was not 32 bit file size limited and officially all OS supported exfat properly.

              Comment


              • #37
                Originally posted by kiffmet View Post
                I hope this answered your questions to a satisfying degree, cheers.
                Yeah that was great thanks! Very informative

                Comment


                • #38
                  Originally posted by jacob View Post

                  Like it or not, the crushing majority uses Windows, which means that everyone else needs to mount Windows filesystems every now and then - whether on dual boot machines, or external hard drives etc. Some people are pragmatic and welcome the prospect of improved interoperability; and some - like you - are members of a cult that makes life more complicated for themselves for no reason at all.
                  You say:
                  1. majority uses Windows
                  2. everyone else needs to mount Windows filesystems every now and then

                  Sorry but you contradict yourself. Why would everyone else, who doesn't run Windows, need to mount Windows filesystems?
                  I don't have any Windows machine at home, why would I need Windows filesystem browsing ability?

                  "some - like you - are members of a cult that makes life more complicated for themselves for no reason at all"
                  How do I complicate life for myself, by not running Windows or by not having any Windows filesystem? For me that's actually very easy life, not needing to interchange between filesystems.

                  Comment


                  • #39
                    Originally posted by piorunz View Post

                    You say:
                    1. majority uses Windows
                    2. everyone else needs to mount Windows filesystems every now and then

                    Sorry but you contradict yourself. Why would everyone else, who doesn't run Windows, need to mount Windows filesystems?
                    I don't have any Windows machine at home, why would I need Windows filesystem browsing ability?

                    "some - like you - are members of a cult that makes life more complicated for themselves for no reason at all"
                    How do I complicate life for myself, by not running Windows or by not having any Windows filesystem? For me that's actually very easy life, not needing to interchange between filesystems.
                    Most people exchange flash drives and other removable storage with people who use other OSes. Many people also use multiple OSes.

                    Comment


                    • #40
                      Originally posted by kiffmet View Post
                      polarathene This patch collection seems to be a byproduct of the custom kernel the repo owner spins for the Archlinux AUR and he has also created custom code & tweaks for this project aswell (ll, lucjan).

                      When you look into the README.md you can see where the patches are coming from. They are formatted in a way that you can selectively add specific features/fixes/patches of i.e. -pf, -ck, …, or archlinux kernels to a vanilla kernel.org sourcetree. A bit more information can be found on the author's gitlab here and here.

                      The patches are mostly organized as diffs between the stable kernel releases (5.13, 5.14, etc.) and the develoment branches of the custom kernel projects (linux-next, ProjectC, linux-pf,… the sources mentioned in the readme). LTS versions may get updated patchsets (feature backports) via this repo for longer aswell. You can get them merged into one file to avoid clutter or separately (-sep) if you want to cherry pick for yourself.

                      Since these development projects (like the ntfs patches or fixes from linux-next) do get updated several times during the lifetime of a stable kernel version or a minor release (i.e. 5.13.X) can make some of the patches obsolete, there are different versions of these patchsets (i.e. ntfs-patches-v1, -v2, -v3 and so on) aswell.

                      Another reason why old versions of the patchsets are still kept there is that stuff might break or not work as expected (they're mostly using out-of-tree code or stuff that hasn't hit linux-stable, afterall) and it allows you to revert to an earlier version.

                      Further you can get, say, an AIO patch containing the latest ProjectC version & dev-branch fixes without having to wait for Alfred to publish a new release as an all-in-one-patch. Be aware that lucjan structured the ProjectC patches differently than Alfred to allow for getting new features in older kernel versions (i.e. prjc-5.14 patchset in 5.13 kernels), meaning that they're incremental updates (require previous -v to be merged aswell), which is not the case for i.e. the ntfs and zstd patches.

                      Alternatively, you can also get the original PDS patches (from before ProjectC) and apply them to a recent kernel. For older kernels, there are also the original BMQ patches. That's the reason why they're listed separately from each other. The original PDS patchset is what was formerly known as "undead PDS" or PDS-mq and was offered a long time for TKglitch custom kernels.

                      The lru patches might be interesting for some very specific users as they are used to build realtime-capable (fully preemptible w/ latency guarantees) kernels.

                      My picked the following patches: I used this repo to get some of the XFS and memory management fixes expected to land with linux-5.15 even before 5.14 stable was released, got a feature that preserves some more memory pages to avoid UI stalling when I use up all my RAM when compiling big stuff, some more block device tuning for SSDs and lower CPU utilization under heavy IO, a security tweak + other misc. stuff like more complete zstd support within the kernel. Then I added ProjectC from Alfred's gitlab (for which I may get updates from lucjan, should I want to stay on 5.13), pulled futex, fsync2 and winesync patches from TKglitch's repo for better Proton support and got the patch for generating cpu-specific optimized kernel code from graysky for a general performance boost.

                      I hope this answered your questions to a satisfying degree, cheers.
                      Amazing explanation. Gonna test the mm fixes, especially the one that protects pages from running software. Also XFS and other bits.

                      Comment

                      Working...
                      X