Kernel 3.0.0 disk io performance drop

[sforum]

Using lvm with two disks I have experienced a significant disk IO performance drop on 3.0.0
compared to 2.6.39.3 more significantly on random write IO.

Has someone else experienced similar?


Kernel 2.6.39.3
Documents/iobenches/2.6.39.3_512M_2G
Iozone: Performance Test of File I/O
Version $Revision: 3.373 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.

Run began: Mon Jul 25 22:51:29 2011

Auto Mode
Include close in write timing
Using Minimum Record Size 1024 KB
Using Maximum Record Size 1024 KB
Using minimum file size of 524288 kilobytes.
Using maximum file size of 2097152 kilobytes.
Command line used: iozone -a -c -y1m -q1m -n512M -g2G -L64 -S3072 -i0 -i2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 3072 Kbytes.
Processor cache line size set to 64 bytes.
File stride size set to 17 * record size.
___________________________ random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
524288 1024 1010151 1552456 4246417 1783257
1048576 1024 653349 1476615 3965058 832824
2097152 1024 345539 357652 3865020 193520

iozone test complete.
In Seconds: 43

Kernel 3.0.0
Documents/iobenches/3.0.0_512M_2G
Iozone: Performance Test of File I/O
Version $Revision: 3.373 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.

Run began: Mon Jul 25 22:57:06 2011

Auto Mode
Include close in write timing
Using Minimum Record Size 1024 KB
Using Maximum Record Size 1024 KB
Using minimum file size of 524288 kilobytes.
Using maximum file size of 2097152 kilobytes.
Command line used: iozone -a -c -y1m -q1m -n512M -g2G -L64 -S3072 -i0 -i2
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 3072 Kbytes.
Processor cache line size set to 64 bytes.
File stride size set to 17 * record size.
__________________________ random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
524288 1024 1017221 1574878 4211487 1817301
1048576 1024 515143 1494402 3842119 212495
2097152 1024 352130 334563 4030001 167668

iozone test complete.
In Seconds: 48

Kernel 2.6.39.3
Documents/iobenches/fio_26393
Flexible IO Tester:
pts/fio-1.1.0 [Test: Intel IOMeter File Server Access Pattern]
Test 1 of 1
Estimated Time Remaining: 3 Minutes
Expected Trial Run Count: 3
Running Pre-Test Script @ 21:12:13
Started Run 1 @ 21:12:13
Running Interim-Test Script @ 21:12:40
Started Run 2 @ 21:12:42
Running Interim-Test Script @ 21:13:06
Started Run 3 @ 21:13:09 [Std. Dev: 1.26%]
Running Post-Test Script @ 21:13:33

Test Results:
23.610690116882
24.161954879761
24.096673965454

Average: 23.96 Seconds (Run Time)


Kernel 3.0.0
Documents/iobenches/fio_300
Estimated Run-Time: 2 Minutes



Flexible IO Tester:
pts/fio-1.1.0 [Test: Intel IOMeter File Server Access Pattern]
Test 1 of 1
Estimated Time Remaining: 2 Minutes
Expected Trial Run Count: 3
Running Pre-Test Script @ 21:02:48
Started Run 1 @ 21:02:48
Running Interim-Test Script @ 21:03:22
Started Run 2 @ 21:03:24
Running Interim-Test Script @ 21:04:24
Started Run 3 @ 21:04:26
Running Interim-Test Script @ 21:04:53 [Std. Dev: 46.97%]
Started Run 4 @ 21:04:55
Running Interim-Test Script @ 21:05:52 [Std. Dev: 40.18%]
Started Run 5 @ 21:05:54
Running Interim-Test Script @ 21:06:20 [Std. Dev: 42.57%]
Started Run 6 @ 21:06:22 [Std. Dev: 39.08%]
Running Post-Test Script @ 21:07:19

Test Results:
30.38679599762
60.021185159683
26.55930685997
57.128906011581
26.059051990509
57.044394016266

Average: 42.87 Seconds (Run Time)

[sforum]

Memory / disk performance drop

Is this maybe the cause?

http://lwn.net/Articles/478440/http://lwn.net/Articles/478440/

[RealNC]

Is this maybe the cause?

http://lwn.net/Articles/478440/http:...ticles/478440/

No, "404 - Not Found" is definitely not the cause of this.

[sforum]

Link to kernel disk performance drop

http://lwn.net/Articles/478440