Announcement

Collapse
No announcement yet.

Diagnosing & fixing unacceptably slow I/O performance

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    That cryptofs is a HUGE overhead, whether the author will admit it or not (the author of such transparent filesystems is usually reluctant to admit that their software is slow, but it is). The overhead is mainly visible when you're reading lots of small files, not one or two big ones. An i3 is a "cheapo" really low-end CPU to be running a cryptofs on, because it's going to take noticeably longer to decrypt data than a faster CPU. I tried a cryptofs on my i7-3820QM and it was so slow that I had to reinstall without it. FWIW my sound menu opens as fast as i can click the button (no noticeable visual delay) without the cryptofs.

    You might intuitively think that most programs installed in /usr or /opt wouldn't read from the encrypted filesystem. You'd be wrong. All the "dot" directories (directories beginning with ".") in your home folder are being very frequently read and written to by various programs; web browsers and gnome programs are particularly egregious offenders. Try running an strace on your favorite slow program and grep for "/home/<user>/."

    A bargain-basement system to begin with is going to really underperform when you throw a lot of crypto at it. I'm not at all surprised. Hardware damage is thrown out by the passed SMART tests, so all I can conclude is that it's slow because it's a slow system. Unless you've been running the exact same software stack (without grabbing updates) for a long time and it was fine but then suddenly started tanking. That would be different.

    My HDD is a HITACHI HTS725050A7E630. The I/O performance doesn't blow me away, but it's certainly about what I'd expect of a typical 7200RPM, 500GB laptop hard drive. Since yours is 5400RPM, latencies will be higher to begin with. Add in the extra effort of a cryptofs and you have a recipe for slow.

    Also -- decrypted pages from a cryptofs are cached in RAM, like any other filesystem, to reduce disk activity. Since Ubuntu greedily eats RAM in the Unity environment, it's not going to leave much RAM left over for page cache. Page cache translates directly to high performance, especially when your fs is encrypted. So your slow hdd has to frequently read lots of tiny little files through an encryption layer on a bargain-basement slow CPU and then read them again when they are accessed again because they're constantly getting bumped out of the page cache.

    Sounds like your problem is a systemic one; i.e., the reason for the observed result isn't just one single component but a combination of factors due to using cheap components and a high-overhead crypto layer. It's kind of like metabolic syndrome in older/obese people, and how your entire body starts to fail, but treating just one organ doesn't slow down the downward spiral. You have to treat the entire system.

    I think you'd have expected performance levels if you:

    1. Double your RAM to at least 4GB (even my extremely cheap employer, who has no idea what IT requirements its employees actually have, gives us 4GB)
    2. Get rid of the ecryptfs
    3. Upgrade your HDD to a 7200rpm model. They have 'em up to at least 750GB but you can find them affordably at the same capacity as you have (500GB).

    I'd almost say that upgrading the processor would help too, except that the CPU isn't going to be nearly as busy if it isn't decrypting files, so you might find that the CPU has plenty of spare cycles when removing the ecryptfs.

    You can also try, without upgrading any hardware, setting your ext4 filesystem to mount as "data=writeback,nobh" (documented here) -- this will give the filesystem much more flexibility about when to commit to disk, which results in less seeking and less frequent disk writes. Writes will pile up in RAM unless specifically fsync()ed to media or unless there's no more room in RAM. It should keep the disk relatively quiet unless it needs to read something that isn't from the page cache (which again is very likely with only 2GB).

    Comment


    • #32
      Power Supply Issues

      Power issues can cause massive hard drive lag. I don't know if SMART would catch that. Just a thought.

      The power supply itself can be an issue, but also molex connections that have been plugged in and out a number of times will loosen up so that the metal parts in the plug won't make good contact -very common issue, but generally associated with system freezes as well as lag.

      We had a computer here that was really bad. It would take like twenty minutes for the hard drive light to settle down when loading winblows. You couldn't do anything for that long, and it was like any button you hit caused the hard drive light to come on full load for minutes and the system wouldn't respond. It was that way for years. Somebody over here just to like complain I think. But it finally broke for good after I did a hard reset on it on one day when I was in a hurry. I hardly ever used that computer, almost never, and there it broke on me. It had been to the shop previously and the smoozer over there didn't even bother to upgrade the 1 GB of RAM, but he did come over and sit on his ass and chatter with not me for well over an hour. I was told it was just so much better after that. Not. Wrong. Nonsense. Turned out to be a power supply issue. It was one of those really, really cheap power supplies. I opened the box to disconnect the hard drive before evaluating the hardware. Couldn't get the optical drive to load a liveCD, really weird activity. Swapped it with a quality power supply and that did it.


      Thanks all.

      Be real, be sober.

      Comment


      • #33
        RAM Drive

        One alternative to that crappy flash memory stuff is a RAM drive. Gigabyte has a bastard product, the I-RAM, which does work in Linux after running the setup in Wine and creating a partition table. It's only 4 GB, but mounting a RAM drive partition in place of offending directories would likely improve the situation. Of course if for some reason the I-RAM battery fails and you completely disconnect the power supply, you'd lose whatever you had on it.

        Oh and I wanted to say, interesting problem, thanks for posting.

        Comment


        • #34
          Thanks

          Originally posted by RealNC View Post
          I specifically mentioned backing up /home. The whole procedure of backup, installing a new distro, and then restoring the previous one takes about 1 hour.
          I've been wondering about that. I'll have to try it just to try it. Simple but critical skill says I. Thanks for posting.

          Comment


          • #35
            Ahh thanks allquixotic, that makes a fair bit of sense all in all. I'll try your ideas.

            I'm a bit unsure about memory, compiz is only using 50mb right now and 800mb is used in total - though perhaps if more was available, it'd cache more as well.

            This is a laptop, so power supply isn't really an issue.
            Last edited by Vadi; 07-16-2012, 12:59 AM.

            Comment


            • #36
              How much hard drive swap for RAM are you actually using? Could you post output from swapon -sv when your experiencing lag. Thanks.


              Code:
              swapon -sv

              Comment


              • #37
                It's really not swapping, it's not out of memory when it's being so slow.

                Code:
                Filename				Type		Size	Used	Priority
                /dev/mapper/cryptswap1                  partition	1820668	29016	-1

                Comment


                • #38
                  Graphics Aperature Size

                  Look at the amount of RAM you have allocated for graphics swap in your system bios. Not sure if that would go to hard drive if you're over the allocation even with RAM available. Maybe the desktop effects is too much and the graphics are using swap on the hard drive. Just a thought.

                  Comment


                  • #39
                    I've looked in there, but it doesn't seem I can change much options at all in regards to that.

                    Comment


                    • #40
                      PDF

                      Best place to look for details is a PDF manual if you can find one. Buy a $30 dollar name brand ATX board off ebay and you likely can find a PDF that details every switch, jumper and other features including the bios, but a pricey laptop....maybe not..

                      Pretty interesting though. Keep us posted.

                      Comment


                      • #41
                        In a twist of my adventures, I apply the data=writeback,nobh option to fstab but not to menu.lst as well. Rebooted - and got a read-only / with a failure to load. Couldn't edit fstab either, because it was read-only as well. Not a good situation to be in when I wasn't quite sure what the eCryptfs passphase was.

                        Managed to remount it with "mount --no-mtab -o remount,rw /dev/sda1 /" and edited the file back, though!

                        Comment


                        • #42
                          I did away with the home folder encryption and performance noticeably improved to the point of being acceptable. Thanks allquixotic for the tip!

                          Comment


                          • #43
                            Originally posted by Vadi View Post
                            I did away with the home folder encryption and performance noticeably improved to the point of being acceptable. Thanks allquixotic for the tip!
                            Cool.

                            yeah, encrypted FS is just starting to be at acceptable perf levels with very new systems (>= 8GB RAM, Sandy/Ivy Bridge) thanks in part to the new crypto extensions (AES-NI) and SSDs. Most people who tried encrypted FS back several years ago on top of the line Core 2 systems with 2 to 4GB RAM (/me *raises hand*) got terrible performance.

                            Comment


                            • #44
                              Originally posted by allquixotic View Post
                              Cool.

                              yeah, encrypted FS is just starting to be at acceptable perf levels with very new systems (>= 8GB RAM, Sandy/Ivy Bridge) thanks in part to the new crypto extensions (AES-NI) and SSDs. Most people who tried encrypted FS back several years ago on top of the line Core 2 systems with 2 to 4GB RAM (/me *raises hand*) got terrible performance.
                              It was not just Linux that had performance issues with encrypted fs's...Windows has some perf issues with full disk encryption as well on such systems

                              Comment


                              • #45
                                Originally posted by Vadi View Post
                                I did away with the home folder encryption and performance noticeably improved to the point of being acceptable. Thanks allquixotic for the tip!
                                Then you can at least create a subdirectory in your /home that will be encrypted for the really important stuff Then basic programs like the file explorer or settings will not need to load from encrypted files.
                                However, then you must be sure the data does not leak out of this subdirectory, e.g. into /tmp or something like /home/.cache/thumbnails*.

                                Comment

                                Working...
                                X