Announcement

Collapse
No announcement yet.

Virtual Data Optimizer (VDO)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Virtual Data Optimizer (VDO)

    Hi,

    Any chance that there will be some tests using the recently opensourced vdo kernel module ?
    It's enables you to use data compression, deduplication and zero block elimination on quite a lot of other filesystems.

    In short vdo creates a dm device that can be used as a block device for other filesystems (or lvm)
    It can be used on a single disk or a md device. (and fiber or whatever you use to store your data on)
    The vdo block device will then do compression deduplication and zero block elimination.

    I was wondering how this would impact performance.
    and maybe seeing how this would work compared to
    btrfs
    xfs on vdo
    xfs on lvm on vdo
    ext4 on vdo
    ext4 on lvm on vdo

    It used to be closed source from permabit, but after red hat bought it they open sourced it and now you can find the git tree here :
    https://github.com/dm-vdo/kvdo

    Cheers
    Tjako

    for reference here are some sources :
    # nice explanation of how it works
    https://rhelblog.redhat.com/2018/02/...rhel-7-5-beta/
    # docs
    https://access.redhat.com/documentat...tion_guide/vdo
    # red hat permabit takeover
    https://www.redhat.com/en/about/pres...ion-technology



  • #2
    Isn't this the lovely thing where if you guess wrong about your data ratio you start to get block write errors if it wasn't able to compress or dedupe enough blocks?

    I think predictability is important. I wouldn't want to run this.

    Comment


    • #3
      Originally posted by Zan Lynx View Post
      I think predictability is important. I wouldn't want to run this.
      Predictability?

      Comment


      • #4
        What do you mean with predictability ?
        The amount of data storage to be used ?
        If so that would be a nice item to test.

        Tjako

        Comment


        • #5
          I heard from a friend who runs Linux servers about problems with a thing that sounds like this VDO.

          The way he told it, basically, you have to tell it ahead of time what compression and deduplication ratio you expect. If you guess incorrectly, everything works fine until it runs out of physical blocks to write to. Then your write returns a write error as if the disk block had gone bad.

          This is not the same as an out of space error. The kernel does not return the error in the same way. In fact, if you have a preallocated data file, as my friend did, and are writing into the middle of it, the failure is unrecoverable because user-space has no idea what write of its caused the problem because of the kernel's file cache and how actual physical writes are asynchronous to the program's writes.

          Comment


          • #6
            I'm not sure that you guys are talking about the same thing.
            When I read the example(first link in the original post) there is no mention off any guesses/estimates regarding the compression and/or de-duplication when setting up the vdo block device.

            T

            Comment


            • #7
              I think it's a different technique, in the example in the first post there is no mention about any estimates related to deduplication or compression.
              You only define the vdo-block devices and then format it to your own likings.

              T

              Comment


              • #8
                You made me go look at https://access.redhat.com/documentat...ating-a-volume

                It's right there at the top. They recommend using ten times logical space for containers and virtual machines and 3x space for other things. Then on one of the following pages they stress the importance of monitoring the vdo and not letting it exceed 80%.

                I note that the documentation there doesn't stress the horrible things that happen when it runs out.

                Comment


                • #9
                  Good point.

                  Didn't see that one yet, that does make this filesystem compression rather dangerous.
                  Even if the system wouldn't be completely lost it is rather 'irritating' to put it mildly that your filesystem corrupts as soon as it fills up.
                  Monitoring the free remaining space is a rather useless advise, cause we all know that shit happens including the failure of the monitor and the filling up of the harddrive when you're not looking.

                  Unless red-hat (the current owner of the vdo module) guarantees no file corruption on a full vdo-block device I won't use it either.

                  T

                  Comment

                  Working...
                  X