Announcement

Collapse
No announcement yet.

IBM Baking Some Nice Optimizations To EXT4's Multi-Block Allocator

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • IBM Baking Some Nice Optimizations To EXT4's Multi-Block Allocator

    Phoronix: IBM Baking Some Nice Optimizations To EXT4's Multi-Block Allocator

    IBM engineers have been working through some multi-block allocator improvements for the EXT4 file-system driver. In particular, they aim to address some shortcomings that were discovered when running various tests on IBM POWER hardware with a 64k block size...

    Phoronix, Linux Hardware Reviews, Linux hardware benchmarks, Linux server benchmarks, Linux benchmarking, Desktop Linux, Linux performance, Open Source graphics, Linux How To, Ubuntu benchmarks, Ubuntu hardware, Phoronix Test Suite

  • #2
    O.o ?!

    Comment


    • #3
      Nice, I have applications where I sometimes have bursts of small file creation, and this could help. :+ )

      Comment


      • #4
        this look great, which filesystem perform well with small files ?

        Comment


        • #5
          Originally posted by luno View Post
          this look great, which filesystem perform well with small files ?
          Historically ReiserFS was good at that, I think this was due to how it was packing files. Which is why I used to use it on Gentoo as it didn't require me to up the inode count on smaller drives like ext fs did as the portage tree is basically massive amount of small files.

          Comment


          • #6
            Originally posted by cb88 View Post

            Historically ReiserFS was good at that, I think this was due to how it was packing files. Which is why I used to use it on Gentoo as it didn't require me to up the inode count on smaller drives like ext fs did as the portage tree is basically massive amount of small files.
            Wait, someone was actually using ReiserFS? That's a first!

            Comment


            • #7
              What is the benefit of using a larger block size than the usual 512b or 4k bytes? is it higher max. throughput?

              Or is it because it is better to match the page size to the block size?
              Last edited by bezirg; 25 May 2023, 11:37 AM.

              Comment


              • #8
                Originally posted by bezirg View Post
                What is the benefit of using a larger block size than the usual 512b or 4k bytes? is it higher max. throughput?

                Or is it because it is better to match the page size to the block size?
                Maximum file system size in 64 bit mode is on the order of 1 yottabytes and in the petabyte rage for 32 bit mode with a block size of 64k with a similar boost to the maximum file sizes. The default block size for ext4fs is 4k, so one shouldn't jump to the conclusion this optimization will have any noticeable effect for anyone not fitting IBM's use case. This is for users with truly massive data stores. (see addendum2)

                Before people start rebuilding their filesystems with 64k block sizes, if you have a massive number of small files, for anything smaller than 64k each file, you just immediately made the on-disk size of your files up to 16x larger over the default 4k. Any difference between the actual data stored and the block size granularity is filled with empty space. Never take these kinds of optimization tests at face value because they're almost always going to address a specific use case. You must test your own uses and balance the costs of an optimization against the actual gain, if any (for example the space lost to block size granularity versus small file size creation, do you really need files larger than 4 TiB and a filesystem larger than 64 ZiB from the default 4k blocks?)

                Edit to add: There are use cases where you'd want different block sizes to improve access times, but these are pretty specific to workload and the underlying storage hardware. Just figured I'd point it out. The average Joe individual Linux user shouldn't need to worry about fine tuning for general desktop use.

                Addendum: Don't confuse physical sector size for the disk with the file system block size. The default block size for ext4fs is 4096 bytes. The usual sector size for drives over 1 TiB is 4k, with consumer grade drives often presenting 512 byte sectors that complicate things for filesystems and volume managers that are 4k+ sector size aware. You might run across some SSDs and giant mechanical drives with sector sizes of 8k+. Again, that's the physical layout, not the filesystem layout. It may or may not improve access performance to match block size with sector size because that's more a necessity of block alignment than block size.

                Addendum2: I was making an assumption I shouldn't have. Power systems seem to move data around in 64k chunks by common default (on AMD64 4k pages is the common default for historical reasons - most software assumes such), so it might make sense that on Power systems that filesystem block sizes are also 64k while ext4fs is most commonly deployed on x86 sooooo... 4k blocks by default. Remember what I said optimal block sizes are workload and hardware dependent?
                Last edited by stormcrow; 25 May 2023, 02:10 PM.

                Comment


                • #9
                  Originally posted by Vistaus View Post

                  Wait, someone was actually using ReiserFS? That's a first!
                  I used it for decades on production servers and was very happy with it.

                  Comment


                  • #10
                    Originally posted by Vistaus View Post

                    Wait, someone was actually using ReiserFS? That's a first!
                    Back in the day, having a 233Mhz Pentium 2, ext3 ate most of the CPU cycles while reiserfs worked like a charm.

                    Comment

                    Working...
                    X