Large HDD/SSD Linux 2.6.38 File-System Comparison

[tytso]

Yeah sure. Didn't know if anybody cared. :P

Some of the things you need to keep very careful of is the data set size is correct for the test.

Like, for example, if I am measuring raw I/O speeds for read/write and I have 4 GB of RAM and the dataset I am working with is only 4-5GB then your not really measuring the file systems as much as measuring the file system cache.

... and if you are testing a copy-on-write or logging file system, such as btrfs or nilfs2, the benchmark needs to write (and delete) at least 2-3 times the free space if you want to fairly test the effects of the garbage collector (in the case of a logging file system) or test the effects of the how well the file system holds up against fragmentation (in the case of copy-on-write file systems, such as ZFS and btrfs).

And of course, if the benchmark isn't testing with the same size files as what you plan to use (small files? typical file sizes as seen by Open Office? MP3 sized files? Video files?), the results may also be not the same.

Finally, for all file systems, how full the file system is during the test will also make a huge difference. It's very easy for file systems to exhibit good performance if you're only using the first 10-20% of the file system. But how does the file system perform when it's 50% full? 75% full?

I can tell you that ext4's performance can be quite bad if you take an ext4 file system which is 1 or 2TB, and fill it up to 95% with file sizes are mostly 8-64 megabytes in size, especially if this happens where many 8-64 meg files are getting deleted, and then replaced with other 8-64 meg files, and the disk gradually fills up until it's 95% full. Since this is a workload (using ext4 as a backend store for cluster file systems) is one that I'm paid to care about at $WORK, I'm currently working on an extension to ext4 to help address this particular case.

Measuring how file systems' performance fall off as the disk utilization increases is hard, yes. But the question is whether the benchmarking is being done primarily for entertainment's sake (i.e., to drive advertising dollars, like a NASCAR race without the car crashes), or to help users make valid decisions about which file system to use, or to help drive improvements in one or more file systems. Depending on your goals, how you approach the file system benchmarking task will be quite different.

-- Ted

[Viper_Scull]

ext4 settings

Michael I read your recently article about why we should leave default settings. I mostly agree with everything, based on the fact that most of the people are gonna use it that way.

But not in this precise scenario: Ext4 + SSD.

90% of the SSD users use Ext4 with noatime and discard options. This is almost mandatory to preserve disk's life. I guess if it were an ssd-mode as in btrfs, this would be as default. so in this case i think default options dont apply here, at least when we talk about number of people using it.

I don't know if results were gonna change much, just wanted to point it out.

[skeetre]

If you would like to recommend a particular set of mount options for each file-system, I would be happy to carry out such tests under those conditions as well to complement the default options.

Michael, if you could, will you benchmark the various file systems with different schedulers/elevators? I'm performing my own comparisons, but I'd like to see noop, cfq, and deadline on the various filesystems compared.

I think this would be a good thing for those of us with multiple types of hard drives to determine which scheduler to use for which hard drive. Should a 2TB use the same as an 80gb drive, or should ssd drives use noop or deadline?

Also, what's the performance difference between relatime, noatime, and atime?

How much difference do these disk benchmarks make depending on your amount of ram? I noticed months ago, part of the reason I went with 12gb instead of 6gb of ram, that the 8gb read tests were WAY higher if you had 8gb or more of ram.

Separate issue... can you make it possible for us to choose colors of each bar on the benchmark graphs? I've done tests where there are 3 systems, and 2 of the 3 have the same color bar, instead of each one having it's own color.

Thanks,

Skeetre

[Haldir]

The color of the EXT4 and JFS bars should be different, it is too similar.

[Jimbo]

If you want a summary of what is the best FS for you to use... use Ext4. It's a safe file system.

JFS is effectively unsupported. It's a port of a file system from OS/2 Warp... the AIX JFS is a entirely different beast. It was interesting when it was new, but besides a few fixes here and there it has essentially been unmaintained for years.

XFS is good if you need big datasets. If you have multiple TB-large file systems then XFS is a good choice. It's fast, it scales well, and it behaves well when dealing with large amounts of data. You'll want to have very good hardware for it... it's not nearly as robust as Ext4 is.

BTRFS is good if you want something to play around with. Otherwise leave it alone until distros start using it by default.

I could not agree more, please don't read this article and begin to think like: hey! I want to use JFS / NILFS2 because I see on phoronix that it has more performance than ext4. Unless you know what are u dealing with, use ext4, it is good, and actively developed by google and Theodore Ts'o.

[popper]

http://openbenchmarking.org/tests/pts

Michael , i meant to ask before but forget, why dont you also mount and run a generic ram disk test , if nothing else it serves as a baseline, and perhaps shows us if there's any DMA regressions in a given test set...

perhaps you should set one up and add it to the results for that machine.

[djchester]

Yeah sure. Didn't know if anybody cared. :P

Great elaboration on the subject drag! I didn't ask for it but sure appreciated it.

[drag]

Care to explain to the _blind_?

Yeah sure. Didn't know if anybody cared. :P


Basically each of those benchmarks mean very different things. Good benchmarks for file systems should include some specific microbenchmark that measures certain characteristics like 'I/O Operations Person Second', 'Throughput', and 'Random Access'. Preferably with a mixture of single thread versus multithread performance.

Some of the things you need to keep very careful of is the data set size is correct for the test.

Like, for example, if I am measuring raw I/O speeds for read/write and I have 4 GB of RAM and the dataset I am working with is only 4-5GB then your not really measuring the file systems as much as measuring the file system cache.

It's extremely easy to get these sorts of benchmarks wrong, and extremely difficult for people to tell if you did them right.

Most of the Phoronix file system benchmarks are like this. From the data on the website it's really impossible to even know what they mean. Based on data sizes, specific options for the benchmarks, and a hundred other variables you could be measuring IOps or random access or kernel file system cache or whatever. It's really difficult to tell.

Then beyond the micro benchmarks that are designed to exercise specific aspects of file systems then you want a number of 'general purpose' application-centric file system benchmarks.

This is the sort of thing that readers here would be more interested in. How long does it take for games to load. How many seconds does it take to go from cold boot to having a browser open and pointing at google. How long does it take for a large spreadsheet get loaded into OpenOffice? How long does it take to do it 300 times with a script?

Then probably you will want to see some latency benchmarks. if your reading audio from a file while doing transcoding how hard can you hit the file system before you start having xruns in jack. Can the file system allow heavy loads, handle multitasking well, give you good performance and yet be responsive? If the max performance of the drive for a single threaded read is 150MB/s and I start reading 30 huge files from the drive and dumping them into /dev/null... does the file system keep chugging along at 150MB/s or does it go into meltdown as it can't handle the load and start thrashing....

What happens when I throw 4 cpus, software raid, and 7 drives at it... does it actually scale any?


Or for server stuff...

With a Apache benchmarks backed by MySQL with a average configuration... how many clients can it support. How long does it take to render a page, how many connections can it handle. Does it scale well? Like if I bump the connections up to a insane level does the file system keep chucking along or does it go into meltdown and not be able to handle all the random I/O in a efficient manner?


Small files, big files, databases.


All this stuff is extremely difficult to do right, time consumer, and worse: hugely expensive.

Which is why nobody really does it. It would take months to put together something proper.


Now what Micheal has done is pretty good for a simple article. The file system devs have more interesting benchmarks, corporate sponsorship, and automated tests, but it's going to be even more difficult for the average user to even understand what is going on.


The thing is is that if you asking for a 'summary' of 'what is best' it's really going to be impossible to tell you.

What are you doing? Video encoding, game playing? Server systems? Are you hosting a moderate site on a VPS with slow storage and mysql... or are you hosting large files? What is your application? what is your goals?

You can't just average all the numbers together and expect to have any meaningful answer. The benchmarks are not all equal... some are better then others, some are more relevant then others. What is important to me may be worthless to you!

Trying to add up all the numbers and giving them different weights and trying to graph out the 'winner' is just silly.

You know how I deal with file systems?

I don't. I just use the defaults and buy gobs of RAM.

Because I know that with a desktop I am not going to be using more then 6-8GB for pretty much anything I'd care to do.

So I buy 16GB. After a month of being up I'll have the entire storage pretty much cache'd in RAM and it'll faster then the fastest SSD. :P

What I care about then is thing like sync, write speeds, and that sort of thing.

For my netbook, however, I only have about 8GB worth of storage. So you know what the best FS for me is on that system? Btrfs. Speed be damned. Why? Because it supports transparent online compression, which works perfectly.

Plus it's not Reiserfs.


If you want a summary of what is the best FS for you to use... use Ext4. It's a safe file system.

JFS is effectively unsupported. It's a port of a file system from OS/2 Warp... the AIX JFS is a entirely different beast. It was interesting when it was new, but besides a few fixes here and there it has essentially been unmaintained for years.

XFS is good if you need big datasets. If you have multiple TB-large file systems then XFS is a good choice. It's fast, it scales well, and it behaves well when dealing with large amounts of data. You'll want to have very good hardware for it... it's not nearly as robust as Ext4 is.

BTRFS is good if you want something to play around with. Otherwise leave it alone until distros start using it by default.

[ayumu]

reiser4

As usual, reiser4 is missing. A shame.

[Jimbo]

I can understand that default options is a fair politic , but in this case is very misleading, the reader with poor knowledge, or the reader who quickly looks for graphs without reading, could obtain wrong ideas about filesystem performance. It's a totally unfair to compare filesystem performance mixing barrier = 0 and barrier = 1 options.

It's true that devs enable some features for some fs, meanwhile others disable them , so totally fair benchmark should be difficult, but at least, with barriers option you should be consistent because it impacts the performance numbers by a lot.