Running An Encrypted LVM In Ubuntu 10.10

[angdraug]

I think it will certainly cause a performance hit if the encryption is CPU bound and not IO bound.

If it is CPU bound, then yes, of course. While it may have been the case with MartjeB's Atom D510 (although to be sure we'd have to see figures of read throughput without encryption on the same system), I still don't see any evidence that the Michael's pgbench test on encrypted LVM was CPU-bound.

As you mentioned yourself, even without AES-NI support in the CPU an i5 should be able to encrypt at ~100 MiB/s, that's much more than 10% of X25-M's read speed, so it doesn't explain an almost tenfold drop in pgbench performance. And I'm sure my dbench tests weren't CPU-bound (my i7 740LM does have AES-NI), and still I see massive performance drop in a write-intensive test on an encrypted volume.

One point I'm ready to concede is that in addition to the corner case I mentioned in my previous post, I see at least two types of modern systems where encrypted LVM performance may be a problem: low-spec netbook/smartbook with a fast SSD, and a cheap NAS with ARM or Atom CPU driving a RAID array of fast SATA disks.

[sreyan]

I don't think CPU load has anything to do with the performance hit on the encrypted volume.

I think it will certainly cause a performance hit if the encryption is CPU bound and not IO bound.

[angdraug]

I don't think CPU load has anything to do with the performance hit on the encrypted volume.

I recently upgraded my laptop from Thinkpad X41 to X201s, going from Pentium-M to i7 and from a rather slow HDD to OCZ Vertex2 SSD, and did a very basic benchmark of both machines using dbench, latex, and glxgears to see how much oomph I've gained.

X41
- tmpfs: Throughput 221.888 MB/sec max_latency=20.515 ms
- ext4: Throughput 7.0933 MB/sec max_latency=1201.097 ms (except Flush: 16.211 ms)
- ext4 aes: Throughput 7.1572 MB/sec max_latency=1494.914 ms (except Flush: 13.813 ms)
- science/tex make: 20.246s
- glxgears: 432.981 FPS

X201s + Vertex 2
- tmpfs: Throughput 765.52 MB/sec max_latency=1.152 ms
- ext4: Throughput 176.659 MB/sec max_latency=251.741 ms (except Flush: 12.441 ms)
- ext4 aes: Throughput 28.3051 MB/sec max_latency=293.534 ms (except Flush: 0.252 ms)
- science/tex make: 7.269s
- glxgears: 1149.474 FPS

On both machines CPU load during disk performance tests was negligible, that's why I'm sure that's not what's slowing down encrypted disk performance on my new laptop. What else can these numbers tell us?

On HDD, there was no difference in performance between plaintext and encrypted volumes, while RAM drive performance shows massive difference between disk and memory thoughput.

Vertex2 SSD is obviously much faster than old 5400rpm HDD (almost as fast as RAM drive on X41), but still nowhere near as fast as RAM drive on X201s. The difference between plaintext and encrypted volume performance is as massive as in pgbench results in the article, but still, my Debian/sid system manages to boot from encrypted root in 13s, which is quite close to what's expected from an SSD drive at its full speed.

If you haven't guessed already, the key differentiator is read vs write operations. The performance difference on write-intensive tests like pgbench and dbench is suspiciously close to the difference between TRIM and non-TRIM modes of operation of SSD drives. And sure enough, because of the way LVM encryption works, it rendrers TRIM useless.

My conclusion:

If you have HDD, use LVM encryption without reservation, it's not going to slow you down at all. If you're doing a lot of write-intesive operations on non-sensitive data and you really need to squeeze every bit of performance out of your SSD, you might want to set aside an unencrypted partition just for that data, encrypting the rest of the system won't cause much lost read performance.

[sreyan]

i720QM doesn't have aes ni

I see you are using an i7. Is that one of the processors with the new AES instructions? I am running a Thinkpad T510 with an i5 that *does* have AES instructions. Since your machine appears to be a Thinkpad of similar vintage, I am going to assume you do.

According to Tom's Hardware a dual core i5 with AES instructions was several times faster than a quad core i7 without. Since these instructions are relatively new, many users won't have them and thus will not have performance numbers quite like yours. It would be nice if you could put a third comparison in there with the aes instructions disabled (I'm not sure if there is a flag for that or if you'd have to rebuild the kernel to disable it).

I think Michael did this test with a i7 720QM. The 720QM is a 45nm "Clarksfield" part, which doesn't have the AES instructions. The 32nm Clarkdale/Arrandale processors have these instructions. There was even some talk at one point that the AES instructions would be implemented on the graphics core included with westmere processors.

Some folks have gotten ~ 550 MiB / sec throughput to ramdrives with a i7-620M (Arrandale). Without AES-NI this drops to ~ 100 MiB / sec. http://www.robo47.net/blog/198-Intel...Debian-Squeeze.

[Markore]

Only trouble is weither your software uses new encryption instructions in processor or not.
Or did you check it out before buying and find Cpu that des not cost much extra cache, like Intel wants.

Anyway, even with a CPU without encryption extensions, in this days cpu speeds, every normal CPU should be able to do just fine with encrypting/decrypting, especially if it have many cores and other cores are used for other cpu-intensive apps, anyway.

So basically, I want to point out that encryption algorithm/application you use to encrypt/decrypt data should be on-pair with hardware you are using.
(Maybe even High-speed hard drive(s) used in test were simply too much throughput etc)

And could also mean that aether there should be changes in a way linux kernel does LVm encryption to be able to fine tune it according to hardware, or what I think is more likely, Database use and needs are not satisfied with current encryption solution, and that is mostly the same.

I am curious how other databases are affected with Linux LVm encryption or maybe to compare it across platforms.

[ChrisIrwin]

I see you are using an i7. Is that one of the processors with the new AES instructions? I am running a Thinkpad T510 with an i5 that *does* have AES instructions. Since your machine appears to be a Thinkpad of similar vintage, I am going to assume you do.

According to Tom's Hardware a dual core i5 with AES instructions was several times faster than a quad core i7 without. Since these instructions are relatively new, many users won't have them and thus will not have performance numbers quite like yours. It would be nice if you could put a third comparison in there with the aes instructions disabled (I'm not sure if there is a flag for that or if you'd have to rebuild the kernel to disable it).

[cynyr]

To give you an idea: I have a small server based on an Atom D510 @ 1.66GHz, which maxes out when reading data from the hard disk. The throughput is ~27 MiB/s.

So using the encryption is fairly CPU intensive. I'm not sure how a D510 compaires to my Athlon X2 BE-2400, but i'm sure the PG test is fairly CPU intensive, xplaining the very very poor results there.

[MartjeB]

Do you have cpu load graphs to go with these benchmarks?

To give you an idea: I have a small server based on an Atom D510 @ 1.66GHz, which maxes out when reading data from the hard disk. The throughput is ~27 MiB/s.

[cynyr]

Do you have cpu load graphs to go with these benchmarks?
Which ones are purely disk based and which are cpu and disk?
Was the encryption overhead starving the cpu in the case of the PG test?

Otherwise thats interesting, can you move normal LVM to encrypted LVM?

[vinc]

Thanks for the article, you just convinced me to encrypt my home folder. By the way, may I ask which is the "more complete solution" you use on your production systems?