ntfs-3g culprit of Ubutu's touted sluggishness (with intensive HDD I/O)?

[Thetargos]

I may have found the source of the touted Ubuntu 8.10 sluggishness with heavy HDD I/O operations ("system slows down to a crawl!" "System stops responding for as long as 1 minute when copying/moving data", etc, etc). Earlier this morning I did a backup of a Windows drive to one of my drives, it wasn't too much, in total about 50 Gigs (copied to my drive, then moved back to another NTFS drive), however when the system was performing the copy, I did see the CPU usage spike quite a bit and the CPU frequency go to top. Though I didn't have any kind of sluggishness or slow down in the desktop, some applications did have a harder time doing some tasks (for example, Firefox took quite a bit longer to render an animated .gif (granted, a fairly large one [5Mb] ), and the speed (usually FF is quite fast at rendering .gifs) was not what I'd expected. So I opened a terminal window and saw in Top that the ntfs-3g process was using about 20-30% CPU (this time I was writing data back to an NTFS drive). Maybe ntfs-3g hammers so hard the kernel for I/O that eventually also affects desktop performance (after all UI operations are also I/O).

At any rate, maybe this would be worth reporting to the ntfs-3g guys or to Ubuntu in particular, maybe is fixable either in fuse or ntfs-3g itself.

By the way I'm using ntfs-3g-1.5012-3 on my Fedora 8 x86_64 box and the reported version is ntfs-3g v2007-08-22-BETA.

[grantek]

I think that kind of CPU usage is normal because the filesystem work is done in user space (ie FUSE, not specifically ntfs-3g). Either way, I've seen it whenever I've used ntfs-3g and it'd be well known - lockups are a different matter though.

[deanjo]

That maybe an overall issue for ubuntu but ntfs-3g shouldn't have been loaded at all on the benchmarks if there was no NTFS volume present.

[Thetargos]

That maybe an overall issue for ubuntu but ntfs-3g shouldn't have been loaded at all on the benchmarks if there was no NTFS volume present.

I'm not speaking of the MacOS benchmarks in this thread, but rather to the fact that many people have complained about system sluggishness/unresponsiveness when under heavy I/O load (like moving data around), most of what I have been able to see is about ntfs volume drives, and I'm not sure if moving data between (say) an ext3 <-> VFAT volume or XFS <-> Ext3 or Ext3 <-> ReiserFS 3/4. All I've seen are people complaints, not many elaborate descriptions of what exactly were the users doing when this perceived unresponsiveness/sluggishness happened.

[Thetargos]

I think that kind of CPU usage is normal because the filesystem work is done in user space (ie FUSE, not specifically ntfs-3g). Either way, I've seen it whenever I've used ntfs-3g and it'd be well known - lockups are a different matter though.

Still, it could condition a resource race condition on some systems where HDD I/O conflicts with X (DRI/DRM) I/O (under extreme circumstances, or maybe another process stealing I/O, like hald?)

[Svartalf]

My issues subsided considerably when switching from ext3 to reiserfs (not to be confused with reiser4...). I was getting off and on grey-out hangs while browsing during a build run up at work- after doing a backup, nuke, and pave of my home partition with reiserfs, it largely went away.

Perhaps there's several things going on, but ext3 has been getting slower over time and it wouldn't be the FIRST time a filesystem caused odd issues on Linux (there was a time gdb wouldn't work right unless you were using ext2 or xfs as the fs the app was executing on- if you used reiserfs, it crashed out.

[Thetargos]

My issues subsided considerably when switching from ext3 to reiserfs (not to be confused with reiser4...). I was getting off and on grey-out hangs while browsing during a build run up at work- after doing a backup, nuke, and pave of my home partition with reiserfs, it largely went away.

Perhaps there's several things going on, but ext3 has been getting slower over time and it wouldn't be the FIRST time a filesystem caused odd issues on Linux (there was a time gdb wouldn't work right unless you were using ext2 or xfs as the fs the app was executing on- if you used reiserfs, it crashed out.

Interesting. I have been a huge fan of ReiserFS, until I started using larger media files that is. It indeed is really fast, especially with small files, but as the files grow bigger, there is not much of a difference with Ext3. The main bummer for me was that my ReiserFS partitions were increasingly prone to corruption (another issue I know is keen to ResierFS 3.x file systems, and supposedly addressed in Reiser4). What I especially liked about ReiserFS was/is its fast scanning when either mount-count is met or when recovering from a power outage, much faster than Ext3. However for the last couple years I have been using Ext3 mostly, and the last partition I used ReiserFS on I formatted it into Ext3 a few days ago (the afore mentioned data corruption probs, and it wasn't the HDD, all other parts in that drive in Ext3 have had no issues). Still I'm puzzled about this seemingly Ubuntu-specific issue with I/O and system hangs, then again, maybe is some upstream component that is causing this? My main rig is still with Fedora 8 (a year-old system now) and have had no issues with it (unlike Fedora 9 on my laptop that has been a never ending source of grief, especially with Xorg crashing all the time). I'll hang in there for bit before updating my system to either Fedora 10 or another distro. I've tested Ubuntu 8.10, and have had a pleasant experience with it, though I'm keen to .rpms rather than .debs (call me a freak).

[deanjo]

though I'm keen to .rpms rather than .debs (call me a freak).

F-R-E-A-K!!! Sorry had to be done. I prefer RPM myself as well.

[Thetargos]

F-R-E-A-K!!! Sorry had to be done. I prefer RPM myself as well.

Nothing to be sorry about, I said call me a freak

[Svartalf]

It indeed is really fast, especially with small files, but as the files grow bigger, there is not much of a difference with Ext3.

Why pick a consistently LOW performer when you can have the high performance for things overall and then have a few gotchas that bring it back down to the low performer's overall level?

The main bummer for me was that my ReiserFS partitions were increasingly prone to corruption (another issue I know is keen to ResierFS 3.x file systems, and supposedly addressed in Reiser4).

That has me a bit concerned to be honest since I've seen occasions where that this is the case. XFS isn't it either- while it's VERY fast, it only journals metadata and has issues on other odd edges as does JFS. I'm kind of keenly waiting to see what comes of btrfs at this point (Not anywhere near ready for primetime- but promises to be at least as good as reiser4 and possibly better than ZFS in many ways...). If Sun hadn't of insisted on the licensing they put on ZFS, this would be mostly a moot point, I think.