Announcement

Collapse
No announcement yet.

ntfs-3g culprit of Ubutu's touted sluggishness (with intensive HDD I/O)?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • ntfs-3g culprit of Ubutu's touted sluggishness (with intensive HDD I/O)?

    I may have found the source of the touted Ubuntu 8.10 sluggishness with heavy HDD I/O operations ("system slows down to a crawl!" "System stops responding for as long as 1 minute when copying/moving data", etc, etc). Earlier this morning I did a backup of a Windows drive to one of my drives, it wasn't too much, in total about 50 Gigs (copied to my drive, then moved back to another NTFS drive), however when the system was performing the copy, I did see the CPU usage spike quite a bit and the CPU frequency go to top. Though I didn't have any kind of sluggishness or slow down in the desktop, some applications did have a harder time doing some tasks (for example, Firefox took quite a bit longer to render an animated .gif (granted, a fairly large one [5Mb] ), and the speed (usually FF is quite fast at rendering .gifs) was not what I'd expected. So I opened a terminal window and saw in Top that the ntfs-3g process was using about 20-30% CPU (this time I was writing data back to an NTFS drive). Maybe ntfs-3g hammers so hard the kernel for I/O that eventually also affects desktop performance (after all UI operations are also I/O).

    At any rate, maybe this would be worth reporting to the ntfs-3g guys or to Ubuntu in particular, maybe is fixable either in fuse or ntfs-3g itself.

    By the way I'm using ntfs-3g-1.5012-3 on my Fedora 8 x86_64 box and the reported version is ntfs-3g v2007-08-22-BETA.

  • #2
    I think that kind of CPU usage is normal because the filesystem work is done in user space (ie FUSE, not specifically ntfs-3g). Either way, I've seen it whenever I've used ntfs-3g and it'd be well known - lockups are a different matter though.

    Comment


    • #3
      That maybe an overall issue for ubuntu but ntfs-3g shouldn't have been loaded at all on the benchmarks if there was no NTFS volume present.

      Comment


      • #4
        Originally posted by deanjo View Post
        That maybe an overall issue for ubuntu but ntfs-3g shouldn't have been loaded at all on the benchmarks if there was no NTFS volume present.
        I'm not speaking of the MacOS benchmarks in this thread, but rather to the fact that many people have complained about system sluggishness/unresponsiveness when under heavy I/O load (like moving data around), most of what I have been able to see is about ntfs volume drives, and I'm not sure if moving data between (say) an ext3 <-> VFAT volume or XFS <-> Ext3 or Ext3 <-> ReiserFS 3/4. All I've seen are people complaints, not many elaborate descriptions of what exactly were the users doing when this perceived unresponsiveness/sluggishness happened.

        Comment


        • #5
          Originally posted by grantek View Post
          I think that kind of CPU usage is normal because the filesystem work is done in user space (ie FUSE, not specifically ntfs-3g). Either way, I've seen it whenever I've used ntfs-3g and it'd be well known - lockups are a different matter though.
          Still, it could condition a resource race condition on some systems where HDD I/O conflicts with X (DRI/DRM) I/O (under extreme circumstances, or maybe another process stealing I/O, like hald?)

          Comment


          • #6
            My issues subsided considerably when switching from ext3 to reiserfs (not to be confused with reiser4...). I was getting off and on grey-out hangs while browsing during a build run up at work- after doing a backup, nuke, and pave of my home partition with reiserfs, it largely went away.

            Perhaps there's several things going on, but ext3 has been getting slower over time and it wouldn't be the FIRST time a filesystem caused odd issues on Linux (there was a time gdb wouldn't work right unless you were using ext2 or xfs as the fs the app was executing on- if you used reiserfs, it crashed out.

            Comment


            • #7
              Originally posted by Svartalf View Post
              My issues subsided considerably when switching from ext3 to reiserfs (not to be confused with reiser4...). I was getting off and on grey-out hangs while browsing during a build run up at work- after doing a backup, nuke, and pave of my home partition with reiserfs, it largely went away.

              Perhaps there's several things going on, but ext3 has been getting slower over time and it wouldn't be the FIRST time a filesystem caused odd issues on Linux (there was a time gdb wouldn't work right unless you were using ext2 or xfs as the fs the app was executing on- if you used reiserfs, it crashed out.
              Interesting. I have been a huge fan of ReiserFS, until I started using larger media files that is. It indeed is really fast, especially with small files, but as the files grow bigger, there is not much of a difference with Ext3. The main bummer for me was that my ReiserFS partitions were increasingly prone to corruption (another issue I know is keen to ResierFS 3.x file systems, and supposedly addressed in Reiser4). What I especially liked about ReiserFS was/is its fast scanning when either mount-count is met or when recovering from a power outage, much faster than Ext3. However for the last couple years I have been using Ext3 mostly, and the last partition I used ReiserFS on I formatted it into Ext3 a few days ago (the afore mentioned data corruption probs, and it wasn't the HDD, all other parts in that drive in Ext3 have had no issues). Still I'm puzzled about this seemingly Ubuntu-specific issue with I/O and system hangs, then again, maybe is some upstream component that is causing this? My main rig is still with Fedora 8 (a year-old system now) and have had no issues with it (unlike Fedora 9 on my laptop that has been a never ending source of grief, especially with Xorg crashing all the time). I'll hang in there for bit before updating my system to either Fedora 10 or another distro. I've tested Ubuntu 8.10, and have had a pleasant experience with it, though I'm keen to .rpms rather than .debs (call me a freak).

              Comment


              • #8
                Originally posted by Thetargos View Post
                though I'm keen to .rpms rather than .debs (call me a freak).
                F-R-E-A-K!!! Sorry had to be done. I prefer RPM myself as well.

                Comment


                • #9
                  Originally posted by deanjo View Post
                  F-R-E-A-K!!! Sorry had to be done. I prefer RPM myself as well.
                  Nothing to be sorry about, I said call me a freak
                  Last edited by Thetargos; 11-14-2008, 09:35 AM.

                  Comment


                  • #10
                    Originally posted by Thetargos View Post
                    It indeed is really fast, especially with small files, but as the files grow bigger, there is not much of a difference with Ext3.
                    Why pick a consistently LOW performer when you can have the high performance for things overall and then have a few gotchas that bring it back down to the low performer's overall level?

                    The main bummer for me was that my ReiserFS partitions were increasingly prone to corruption (another issue I know is keen to ResierFS 3.x file systems, and supposedly addressed in Reiser4).
                    That has me a bit concerned to be honest since I've seen occasions where that this is the case. XFS isn't it either- while it's VERY fast, it only journals metadata and has issues on other odd edges as does JFS. I'm kind of keenly waiting to see what comes of btrfs at this point (Not anywhere near ready for primetime- but promises to be at least as good as reiser4 and possibly better than ZFS in many ways...). If Sun hadn't of insisted on the licensing they put on ZFS, this would be mostly a moot point, I think.

                    Comment


                    • #11
                      Originally posted by Svartalf View Post
                      Why pick a consistently LOW performer when you can have the high performance for things overall and then have a few gotchas that bring it back down to the low performer's overall level?
                      Well, for some reason (despite its performance, which for me has been rather good... for my needs that is) Ext3 is the best supported filesystem across distributions, not to mention is touted to be "Linux's native file system". I really do not care that much about a filesystem. All I want is decent enough access times to write and fetch my data, and my data to remain uncorrupted as much as possible (since I'm a physician, you can imagine this is of especial interest for me, I don't want any patient's records to be corrupted [which also prompts for rather frequent backups]).


                      Originally posted by Svartalf View Post
                      That has me a bit concerned to be honest since I've seen occasions where that this is the case. XFS isn't it either- while it's VERY fast, it only journals metadata and has issues on other odd edges as does JFS. I'm kind of keenly waiting to see what comes of btrfs at this point (Not anywhere near ready for primetime- but promises to be at least as good as reiser4 and possibly better than ZFS in many ways...). If Sun hadn't of insisted on the licensing they put on ZFS, this would be mostly a moot point, I think.
                      I had been playing with the idea of converting my partitions to XFS, but as you point out, it doesn't implement a full journal, which kind of concerns me... However, AFAIK is the preferred FS for media content-creation as it has unsurpassed I/O performance with especially big files (movie raw data files?). I wish Sun did allow for a more flexible license on their ZFS (they know that by itself it is a GREAT asset ), and have been off the loop in the recent developments of other "better" FSes, so I don't know "what's out there". I know Ext4 debuted in one of the 2.6.2x kernels, but I do not know what exactly does have over Ext3, or its performance, etc. For the time being, I'm happy with what I've got, though.

                      Comment


                      • #12
                        Originally posted by Thetargos View Post
                        Well, for some reason (despite its performance, which for me has been rather good... for my needs that is) Ext3 is the best supported filesystem across distributions, not to mention is touted to be "Linux's native file system". I really do not care that much about a filesystem. All I want is decent enough access times to write and fetch my data, and my data to remain uncorrupted as much as possible (since I'm a physician, you can imagine this is of especial interest for me, I don't want any patient's records to be corrupted [which also prompts for rather frequent backups]).
                        You should actually care about most of the piece-parts of your OS install. The corrupted records is of concern. Oh, and "best supported" doesn't mean anything other than people poring over the code and "fixing" things- it does mean brown paper bag over the head problems get found quickly, but it doesn't go that it's the best choice (Keep in mind- the thinking, even from the main kernel crowd, is that if btrfs comes in as good as it's still looking it'll replace everything else as the mainline filesystem... And it's got things that may leapfrog ZFS, which IS one of the best filesystem designs to date...).

                        What does it matter if my FS is "robust" if I can't use the box for minutes at a time because I'm doing something disk-intensive... Is the answer jump to a new distribution? Not always. There's other...issues...that are not unlike this one with Ubuntu in Fedora, Mandriva, etc... Every one of them make dumb decisions from time to time.



                        I had been playing with the idea of converting my partitions to XFS, but as you point out, it doesn't implement a full journal, which kind of concerns me... However, AFAIK is the preferred FS for media content-creation as it has unsurpassed I/O performance with especially big files (movie raw data files?). I wish Sun did allow for a more flexible license on their ZFS (they know that by itself it is a GREAT asset ), and have been off the loop in the recent developments of other "better" FSes, so I don't know "what's out there". I know Ext4 debuted in one of the 2.6.2x kernels, but I do not know what exactly does have over Ext3, or its performance, etc. For the time being, I'm happy with what I've got, though.
                        Ext4 MIGHT be okay, but unfortunately, you can't boot with it right at the moment- however, it may have the same glitch that Ext3 currently has. XFS and JFS don't do anything other than metadata journalling, but in and of itself with a UPS, you should mostly be fine with them- and you CAN boot with either of those or Reiserfs (though we know what gives with this...I probably need to move to XFS or JFS on my home dir here at work...). Btrfs is in development and is expected to be mostly ready for prime-time in 6 or so months. There's issues with it that are currently unresolved (as in not fully implemented yet...) so I wouldn't use it for anything important. ZFS is available as a fuse filesystem and could be a choice for some things.
                        Last edited by Svartalf; 11-14-2008, 12:23 PM.

                        Comment


                        • #13
                          Forgive me if I'm stating the obvious here, but couldn't it just be that the kernel is not tuned for desktop? Last time I looked at the Ubuntu Kernel sources the I/O latency settings were set to what was essentially the server option

                          Comment

                          Working...
                          X