Announcement

Collapse
No announcement yet.

Real World Benchmarks Of The EXT4 File-System

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Hmm this makes me want to try XFS again but only if my drives support write barriers. I've just been trying to find a way to determine whether the drive supports them but I can't find one anywhere. Is there any way besides actually creating an XFS partition?

    Comment


    • #32
      I'd like to see boot time, app start time, package manager, and maybe file indexer/searching service benchmarks.

      Configuring each filesystem with settings commonly recommended by the community and including Reiser4 and JFS would be very interesting too.

      Comment


      • #33
        I have XFS filesystem on my / partition on the desktop computer mounted with nobarrier option - I've been told this option is dangerous, but I haven't experienced any filesystem corruption since then, even with few unexpected reboots. I look forward to trying out the little XFS tweaks from this thread ^^

        I use ext4dev on my laptop, and it's working normally since 2.6.26 - I had some problems with block allocation on 2.6.25, but no files were corrupted or lost.

        Comment


        • #34
          Originally posted by kebabbert View Post
          ext4 seems cool! Does it protect against silent corruption? Typically 20% of a modern hard drive is devoted to error correcting codes. Once in a while, you will run into a problem that is not correctable, or what is worse; not detectable. You dont even know that there was some error in your files:
          http://www.acmqueue.org/modules.php?...owpage&pid=504

          And Ive heard about a large ext3 filesystem being fsck, it took one week. Does ext4 suffer from the same problem?
          No, ext4 does not. The only filesystems that can detect data corruption are ZFS and BtrFS currently.

          Comment


          • #35
            LINUX FILESYSTEM BENCHMARKS
            (includes Reiser4 and Ext4)


            http://linux.50webs.org/
            http://m.domaindlx.com/LinuxHelp/
            http://linuxhelp.150m.com/

            RESULT: With compression, REISER4, absolutely SMASHED the other filesystems.

            No other filesystem came close (not even remotely close).

            Using REISER4 (gzip), rather than EXT2/3/4, saves you a truly amazing 816 - 213 = 603 MB (a 74% saving in disk space), and this, with little, or no, loss of performance when storing 655 MB of raw data. In fact, substantial performance increases were achieved in the bonnie++ benchmarks.

            We use the following filesystems:

            REISER4 gzip: Reiser4 using transparent gzip compression.
            REISER4 lzo: Reiser4 using transparent lzo compression.
            REISER4 Standard Reiser4 (with extents)
            EXT4 default Standard ext4.
            EXT4 extents ext4 with extents.
            NTFS3g Szabolcs Szakacsits' NTFS user-space driver.
            NTFS NTFS with Windows XP driver.

            Disk Usage in megabytes. Time in seconds. SMALLER is better.

            Code:
            .-------------------------------------------------.
            |File         |Disk |Copy |Copy |Tar  |Unzip| Del |
            |System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
            |Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
            .-------------------------------------------------.
            |REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
            |REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
            |REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
            |REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
            |NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
            |NTFS         | 779 | 781 | 173 |   X |   X |   X |
            |REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
            |XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
            |JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
            |EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
            |EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
            |EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
            |EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
            |FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
            .-------------------------------------------------.
            WHAT THE NUMBERS MEAN:

            The raw data (without filesystem meta-data, block alignment wastage, etc) was 655MB.
            It comprised 3 different copies of the Linux kernel sources.

            Disk Usage: The amount of disk used to store the data.
            Copy 655MB (1): Time taken to copy the data over a partition boundary.
            Copy 655MB (2): Time taken to copy the data within a partition.
            Tar Gzip 655MB: Time taken to Tar and Gzip the data.
            Unzip UnTar 655MB: Time taken to UnGzip and UnTar the data.
            Del 2.5 Gig: Time taken to Delete everything just written (about 2.5 Gig).

            Each test was preformed 5 times and the average value recorded.

            To get a feel for the performance increases that can be achieved by using compression, we look at the total time (in seconds) to run the test:

            bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)

            Code:
            .-------------------.
            | FILESYSTEM | TIME |
            .-------------------.
            |REISER4 lzo |  1938|
            |REISER4 gzip|  2295|
            |REISER4     |  3462|
            |EXT4        |  4408|
            |EXT2        |  4092|
            |JFS         |  4225|
            |EXT3        |  4421|
            |XFS         |  4625|
            |REISER3     |  6178|
            |FAT32       | 12342|
            |NTFS-3g     |>10414|
            .-------------------.
            The top two results use Reiser4 with compression. Since bonnie++ writes test files which are almost all zeros, compression speeds things up dramatically. That this is not the case in real world examples can be seen in the first test above where compression often does not speed things up. However, more importantly, it does not slow things down much, either.

            http://linux.50webs.org/
            http://linuxhelp.150m.com/resources/fs-benchmarks.htm
            http://m.domaindlx.com/LinuxHelp/res...benchmarks.htm
            Last edited by Jade; 12-03-2008, 11:20 PM.

            Comment


            • #36
              A couple points:

              1. Reiserfs certainly isn't the first FS to support online compression. I remember there being compressed file system options going back to DOS days.

              2. Most big files that everybody cares about... things like:

              * audio files
              * image files
              * document files (like OO.org's)
              * video files
              * game archives


              Are already compressed prior to being written to disk. Usually they use specialized compression that is much better for that specific format then generic ones like Gzip or Lzo. Ergo for anything that is especially sensitive to I/O speeds or are large are going to gain very little to no benefit from compression at the file system level.

              So most the benefit you'll gain will be from large text files and executables. Maybe from certain types of database actions.

              So you can get some boosts in performance in terms of application start up and whatnot. But the killer for startup performance is mostly seek times, not read limitations.. which compression does nothing to address and may actually make worse. Likely make latency worse.

              Having optional online compression support is certainly a good thing, (I would love to have it) but it's no homerun hit like those benchmarks misleadingly seem to indicate.

              Oh an point 3:

              Reiserfs is bad for your data's health.

              --------------------------------------

              BTW, Btrfs has transparent compression support as one of it's listed features.

              Comment


              • #37
                point 3 is so wrong. reiserfs (3.X) had some problems in early 2.4 because of constant breakage introduced by vm changes (the in kernel code will be fixed when it is affected lie). But later 2.4 and 2.6 reiserfs is one of the most stable fs out there. Only bug fixes, no features. Just look on lkml. Weekly XFS bugs, monthly ext3 bugs, once in a while a non-reproducible reiserfs bug.
                And reiser4 already works better for me than ext3 ever did.

                Comment


                • #38
                  Originally posted by Jade View Post
                  LINUX FILESYSTEM BENCHMARKS
                  (includes Reiser4 and Ext4)
                  Sorry, let me clarify. I'd like to see filesystem benchmarks that include current versions of Reiser4, ext4, and so on all configured as commonly recommended by their fans.

                  Comment


                  • #39
                    fsck time

                    Hi,

                    first of all, thanks for the articel and benchmark.

                    We are planning to buy a new raid system with around 4TB of storage capacity (actual we have 2TB on ext3). On monthly scheduled administration days we reboot the main server for maintenance (new kernel, surely kick all nfs clients ...). So, from time to time, the raid system will check (tunefs could avoid this, but for safety reasons we perform the complete disk check) the data. This needs hours where you just can wait and wait ....

                    So, if ext4 would reduce this checking time, i would immediatley change.

                    Any experiences or a possibility to check this???

                    Thanks in advance

                    Comment


                    • #40
                      Originally posted by energyman View Post
                      point 3 is so wrong. reiserfs (3.X) had some problems in early 2.4 because of constant breakage introduced by vm changes (the in kernel code will be fixed when it is affected lie). But later 2.4 and 2.6 reiserfs is one of the most stable fs out there. Only bug fixes, no features. Just look on lkml. Weekly XFS bugs, monthly ext3 bugs, once in a while a non-reproducible reiserfs bug.
                      And reiser4 already works better for me than ext3 ever did.

                      Reiser and friends refused to support and maintain Reiserfsv3 once they started work on Reiserfsv4. All file systems are very complex and have problems that are going to crop up over time, refusing to support it means that bugs and problems went unfixed, unacknowledged, and forced other developers to pick up their work if it happened at all.

                      Rfsv4 isn't ever going to get into the kernel at this point. At least not in any substantial way. It has no serious backing from the core Linux developers or any of the corporations or distributions that depend on Linux. Without this testing and support it's never going to mature to the point were I am going to trust it.

                      If Reiser was able to relate and get along with other people then it probably would of had a good chance, but this is just not how it worked out.

                      And speed, btw, is extremely secondary to data protection and reliability.

                      Ask yourself why, if companies like Redhat or IBM, that depend on the competitive nature of Linux against operating systems like Solaris and Windows 2008, have continued to put substantial time and effort into maintaining Ext3 and evolving it to Ext4 instead of just adopting the 'much better' Reiserfsv4.

                      Suse was a early adopter and proponent of ReiserFsv3. They have ReiserFS developers on staff. They show no sign of moving to support v4 in any meaningful way. They too depend heavily on the ability of Linux to compete with Unix, Windows, and especially Redhat. So you would think that if v4 offered a substantial advantage over the more mundane Linux file systems then they would jump at the chance to push their OS forward.

                      And why Intel, HP, IBM, Oracle, and Redhat are currently spending a great deal of money, developer time, and expertise working and hacking on Btrfs. People who make their living designing and developing file systems... things like Ext2/3, Advfs, Ocfsv2, Reiserfs, Lustre, GFS, and XFS.

                      Because they have very good reasons for this sort of thing and understanding why is critical to really knowing how to gauge file systems. Now they are not gods, are imperfect, with biases, and lots of decisions are driven by politics as much as technical reasons, but there is method behind all this madness.


                      -------------------------------


                      For example:

                      Ext3 fsck is something that is rather powerful at protecting and repairing file systems. It's robust, proven, and reliable.

                      Ext3 journalling also has the capabilities to monitor and detect issues with actual files. Sure it's not as good as checksums, but it does have the ability to do journalling for files. Now other journalling FSs, like Reiser and XFS only monitor file system metadata (that is they monitor the FS and ignore the data store using the FS) and that is all they are capable of dealing with. (unfortunately with Ext3 there are bugs for a long time that tended to limit the usefulness of this feature)

                      If you run Reiserfsck on a file system with a file with a loopback file with Reiserfs on it.. then it will often try to stitch your loopback file into the file system. Which is the sort of thing you very much do not want.

                      That's a problem with v3 that was never fixed. It isn't suppose to be a problem v4, but keep in mind that unlike Ext2->Ext3->Ext4 each new Reiser file system is rewritten from scratch and are not related to one another in any direct manner.
                      Last edited by drag; 12-04-2008, 02:41 AM.

                      Comment


                      • #41
                        Originally posted by Jade View Post
                        RESULT: With compression, REISER4, absolutely SMASHED the other filesystems.
                        Oh dear.. Well first off how can I say this.. You just made me CHOKE on my coffee. Haha, you know, the only time I used reiserFS, it was a bad experience, eventually . So even if it is faster, its definitely not as proven or as reliable as something like EXT3. I personally wouldn't be surprised to see ReiserFS3 be removed from the Linux Kernel eventually. Because from my experience at least, and what I've heard from others, its really not that good.

                        Comment


                        • #42
                          Originally posted by drag View Post
                          Suse was a early adopter and proponent of ReiserFsv3. They have ReiserFS developers on staff. They show no sign of moving to support v4 in any meaningful way. They too depend heavily on the ability of Linux to compete with Unix, Windows, and especially Redhat. So you would think that if v4 offered a substantial advantage over the more mundane Linux file systems then they would jump at the chance to push their OS forward.
                          Just to add on:

                          Suse dropped Reiser as it's default filesystem because of several technical problems, as well as problems related to maintenance especially after Chris Mason left (the people basically left holding th bag on maintaining it). That left basically Mahoney to look after it and with it's bug ridden past it just became to big of a headache. It also wasn't so shit hot in performance or reliabilty as well. I wouldn't be surprised if it is soon dropped from the supported filesystems all together in suse.

                          ReiserFS has no future. It's effectively dead. Time to put it up on the shelf with other innovations like the Superdisk 120 and the 80186.

                          Comment


                          • #43
                            Which bit of this didn't you understand?

                            LINUX FILESYSTEM BENCHMARKS
                            (includes Reiser4 and Ext4)


                            http://linux.50webs.org/
                            http://m.domaindlx.com/LinuxHelp/
                            http://linuxhelp.150m.com/

                            Some Amazing Filesystem Benchmarks. Which Filesystem is Best?
                            http://www.phoronix.com/forums/showthread.php?t=1765

                            RESULT: With compression, REISER4, absolutely SMASHED the other filesystems.

                            No other filesystem came close (not even remotely close).

                            Using REISER4 (gzip), rather than EXT2/3/4, saves you a truly amazing 816 - 213 = 603 MB (a 74% saving in disk space), and this, with little, or no, loss of performance when storing 655 MB of raw data. In fact, substantial performance increases were achieved in the bonnie++ benchmarks.

                            We use the following filesystems:

                            REISER4 gzip: Reiser4 using transparent gzip compression.
                            REISER4 lzo: Reiser4 using transparent lzo compression.
                            REISER4 Standard Reiser4 (with extents)
                            EXT4 default Standard ext4.
                            EXT4 extents ext4 with extents.
                            NTFS3g Szabolcs Szakacsits' NTFS user-space driver.
                            NTFS NTFS with Windows XP driver.

                            Disk Usage in megabytes. Time in seconds. SMALLER is better.

                            Code:
                            .-------------------------------------------------.
                            |File         |Disk |Copy |Copy |Tar  |Unzip| Del |
                            |System       |Usage|655MB|655MB|Gzip |UnTar| 2.5 |
                            |Type         | (MB)| (1) | (2) |655MB|655MB| Gig |
                            .-------------------------------------------------.
                            |REISER4 gzip | 213 | 148 |  68 |  83 |  48 |  70 |
                            |REISER4 lzo  | 278 | 138 |  56 |  80 |  34 |  84 |
                            |REISER4 tails| 673 | 148 |  63 |  78 |  33 |  65 |
                            |REISER4      | 692 | 148 |  55 |  67 |  25 |  56 |
                            |NTFS3g       | 772 |1333 |1426 | 585 | 767 | 194 |
                            |NTFS         | 779 | 781 | 173 |   X |   X |   X |
                            |REISER3      | 793 | 184 |  98 |  85 |  63 |  22 |
                            |XFS          | 799 | 220 | 173 | 119 |  90 | 106 |
                            |JFS          | 806 | 228 | 202 |  95 |  97 | 127 |
                            |EXT4 extents | 806 | 162 |  55 |  69 |  36 |  32 |
                            |EXT4 default | 816 | 174 |  70 |  74 |  42 |  50 |
                            |EXT3         | 816 | 182 |  74 |  73 |  43 |  51 |
                            |EXT2         | 816 | 201 |  82 |  73 |  39 |  67 |
                            |FAT32        | 988 | 253 | 158 | 118 |  81 |  95 |
                            .-------------------------------------------------.
                            WHAT THE NUMBERS MEAN:

                            The raw data (without filesystem meta-data, block alignment wastage, etc) was 655MB.
                            It comprised 3 different copies of the Linux kernel sources.

                            Disk Usage: The amount of disk used to store the data.
                            Copy 655MB (1): Time taken to copy the data over a partition boundary.
                            Copy 655MB (2): Time taken to copy the data within a partition.
                            Tar Gzip 655MB: Time taken to Tar and Gzip the data.
                            Unzip UnTar 655MB: Time taken to UnGzip and UnTar the data.
                            Del 2.5 Gig: Time taken to Delete everything just written (about 2.5 Gig).

                            Each test was preformed 5 times and the average value recorded.

                            To get a feel for the performance increases that can be achieved by using compression, we look at the total time (in seconds) to run the test:

                            bonnie++ -n128:128k:0 (bonnie++ is Version 1.93c)

                            Code:
                            .-------------------.
                            | FILESYSTEM | TIME |
                            .-------------------.
                            |REISER4 lzo |  1938|
                            |REISER4 gzip|  2295|
                            |REISER4     |  3462|
                            |EXT4        |  4408|
                            |EXT2        |  4092|
                            |JFS         |  4225|
                            |EXT3        |  4421|
                            |XFS         |  4625|
                            |REISER3     |  6178|
                            |FAT32       | 12342|
                            |NTFS-3g     |>10414|
                            .-------------------.
                            The top two results use Reiser4 with compression. Since bonnie++ writes test files which are almost all zeros, compression speeds things up dramatically. That this is not the case in real world examples can be seen in the first test above where compression often does not speed things up. However, more importantly, it does not slow things down much, either.

                            http://linux.50webs.org/
                            http://linuxhelp.150m.com/resources/fs-benchmarks.htm
                            http://m.domaindlx.com/LinuxHelp/res...benchmarks.htm
                            Last edited by Jade; 12-04-2008, 05:23 AM.

                            Comment


                            • #44
                              Originally posted by drag View Post
                              but keep in mind that unlike Ext2->Ext3->Ext4 each new Reiser file system is rewritten from scratch and are not related to one another in any direct manner.
                              Then again, ext4 with extents is not compatible with ext3 or ext2 at all. You cannot ever never mount a fully featured ext4 fs as them. I actually found it weird this benchmark lacked fsck tests for ext3 and ext4. The speedup in those is the main thing I'm looking forward into. My conclusions from the benchmarks would be that ext4 excels with big files, probably due to the new extents, but performance difference otherwise is not significant. Performance differences would likely have been smaller with smaller test files. Maybe the other tests didn't deal with gigabytes of data? "Extents are introduced to replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation. A single extent in ext4 can map up to 128MiB of contiguous space with a 4KiB block size." Wikipedia. And yeah, you need to fully reformat the hard disk to get full benefits of ext4 afaik. (That is, extents for old files too)

                              Comment


                              • #45
                                Originally posted by mctop View Post
                                Hi,

                                first of all, thanks for the articel and benchmark.

                                We are planning to buy a new raid system with around 4TB of storage capacity (actual we have 2TB on ext3). On monthly scheduled administration days we reboot the main server for maintenance (new kernel, surely kick all nfs clients ...). So, from time to time, the raid system will check (tunefs could avoid this, but for safety reasons we perform the complete disk check) the data. This needs hours where you just can wait and wait ....

                                So, if ext4 would reduce this checking time, i would immediatley change.

                                Any experiences or a possibility to check this???

                                Thanks in advance
                                Have you tried Solaris and ZFS? ZFS has no fsck. Instead, it has something called "scrub" but your ZFS raid is online and fully functioning mean while. Ive heard that to fsck a large ext3 took one week!

                                Here is a Linux admin comparing ZFS with linux filesystems:
                                http://lethargy.org/~jesus/archives/...ver-Linux.html

                                Here is a Linux guy setting up a home file server ZFS:
                                http://breden.org.uk/2008/03/02/a-ho...ver-using-zfs/

                                ZFS + 48 SATA discs + dual Opteron and no hardware raid (just plain SATA controller), writes more than 2 GB/sec:
                                http://milek.blogspot.com/2006/11/th...hroughput.html

                                And SUN is selling a new Storage device, 7000. Read about "The Killer App". You could download and play with that analysis software that uses unique DTrace in a VMware image (which simulates several discs with ZFS raid):
                                http://blogs.sun.com/mws/entry/intro...n_storage_7000




                                Create a ZFS raid:
                                # zpool create myZFSraid disc0 disc1 disc2 disc3
                                and that is all. No formatting needed, just bang away immediately. Dead simple administration.
                                Last edited by kebabbert; 12-04-2008, 06:16 AM.

                                Comment

                                Working...
                                X