Linux 3.14 File-System HDD Benchmarks

[V10lator]

4) then dd the drive, piping THAT into tar
5) and tell it to compress it.

I know this is off-topic but is there any reason for tar? AFAIK you could pipe the dd output directly to the compressing program instead of using the (useless as it's just one file) container.

[GreatEmerald]

I get that sometimes you WANT to do dd because you need a PERFECT bit by bit copy with zero possible errors... But even compressing it, I really doubt that you'll save much space unless you first zero out the drive, which means we need something better and smarter than just sector by sector.

I assume that Acronis does all that by checking NTFS metadata to see what space is occupied and what space isn't? Because, again, the same thing is possible with btrfs send/receive, you'll get a small snapshot with only the data you need. The other partitions, well, you'd have to use some similar filesystem tools in those. I can see how this would be doable, but I can also see how it could fail horribly (you'd need to handle MBR, partition flags, UEFI partitions, suspend partitions, OEM partitions, MS hidden partitions, etc. etc. and all of the filesystems on them).

As for simple reinstalls, we have deployment ISOs (see SUSE Studio) for that. You insert a CD/USB drive, boot from it, and the image gets deployed on the hard drive. Much safer than to make a disk copy and pray that copying it back makes everything work as expected.

[Ericg]

I know this is off-topic but is there any reason for tar? AFAIK you could pipe the dd output directly to the compressing program instead of using the (useless as it's just one file) container.

Most How-To's and such use Tar, so I went with tar. But AFAIK there is not anything stopping you from piping it directly to like lzo or 7z. I was just using the steps that i had seen the most often over the last month or so of trying to find a linux-native way of doing it.

[Ericg]

I assume that Acronis does all that by checking NTFS metadata to see what space is occupied and what space isn't? Because, again, the same thing is possible with btrfs send/receive, you'll get a small snapshot with only the data you need. The other partitions, well, you'd have to use some similar filesystem tools in those

AFAIK, Yes, Acronis just uses NTFS utilities to query the metadata and pick whats needed. And yes the same thing is possible with Btrfs.. but that requires btrfs on both drives. Unfortunately there's no utilities out there right now that actually DO use the filesystem utilities and API's to figure out whats going on inside the filesystem. PartImage, PartClone, (AFAIK) CloneZilla, Gnome-Disks, Gparted, all just take the easy route and do a sector-by-sector copy. Hell they may just be frontends for dd anyway.

The situation at the shop is we have 1 external hard drive that is an "Image drive" which has a directory layout of images and we have an Acronis True Image liveCD. Fire up the LiveCD and do everything in a limited Acronis environment and then everything is in Acronis' secret format.

What I was hoping for was throwing Fedora on my external hard drive, installed, and whenever I needed to image a drive I would plug it in and boot onto the installed, full, Fedora system on the drive and all drive images would be in a ~/Drive Images folder and all client data backups would be under ~/Client Data. Images would (hopefully) be taken / restored from linux native utilities from within Fedora... but I can't do that if every image is multi-hundred gigabytes.

So I'm back to the way we do it at the shop with Acronis Live CD's because its infinitely more space efficient.

As for simple reinstalls, we have deployment ISOs (see SUSE Studio) for that. You insert a CD/USB drive, boot from it, and the image gets deployed on the hard drive. Much safer than to make a disk copy and pray that copying it back makes everything work as expected.

I'll check out Suse Studio for Suse installs, but that doesn't help me for XP or Win7 installs where every model has different drivers needed and may have different patch-levels installed.

[GreatEmerald]

AFAIK, Yes, Acronis just uses NTFS utilities to query the metadata and pick whats needed. And yes the same thing is possible with Btrfs.. but that requires btrfs on both drives.

I don't think so, I don't see why you couldn't loop-mount a Btrfs volume inside a file for the purposes of receive (though I've never tried it, so I'm not really sure if it works in practise).

Unfortunately there's no utilities out there right now that actually DO use the filesystem utilities and API's to figure out whats going on inside the filesystem. PartImage, PartClone, (AFAIK) CloneZilla, Gnome-Disks, Gparted, all just take the easy route and do a sector-by-sector copy. Hell they may just be frontends for dd anyway.

Yeap. It would require quite a bit of effort to make such an utility, as you might expect (and it seems that for the most part people are fine with the whole zero out/dd/bzip thing; at least that's the official method of compacting VirtualBox disk images, for instance). Of course, there is nothing stopping you from trying to code something of the sort on your own That's the usual workflow of making new utilities that nobody thought about earlier appear. That's also how I ended up packaging Snapper (which is openSUSE tech) for Gentoo

So I'm back to the way we do it at the shop with Acronis Live CD's because its infinitely more space efficient.

Lies! It's probably around 5 times more efficient on average

I'll check out Suse Studio for Suse installs, but that doesn't help me for XP or Win7 installs where every model has different drivers needed and may have different patch-levels installed.

Well, with Windows, you have unattended install CDs for pretty much the same purpose. You can use nLite for Windows XP for that. For Win7, there isn't a universally accepted good ISO customisation utility, but there are a few different ones with different pros and cons. All of the utilities allow you to customise what drivers are included in the ISO by editing a config file. I use these utilities to set up ultra-minimal Windows images for use on virtual machines (for software that doesn't run under Wine), for instance.

[gilboa]

Yes. In fact, I have a LiveCD build on SUSE Studio just for that (it automatically pulls in the latest kernel and btrfs tools for performing complex offline tasks; last time I used it to set up a backup in a different drive using btrfs-send/btrfs-receive). The btrfs check tool itself I usually run regularly to see if it reports any problems. There sometimes are some problems when I have to force-poweroff the machine (like when the GPU hangs), but they usually amount to some warnings and some data truncation. The former are mostly solved by running btrfs scrub. There was a case when it didn't help, and I needed to run the btrfs check repair to solve those. (There was also a time when btrfs check reported a lot of false positives, and even the warnings emitted now generally don't amount to much ? the developers themselves can't tell what some of them are about without having direct access to the drive in question). There was also one time some years ago when I had bad RAM, which caused corruption that btrfs check repair couldn't solve either, but that's pretty much a given (and I still was able to retrieve all the data I needed).

Thanks.
I guess its time to start playing with btrfs.

- Gilboa

[gadnet]

test of lvm/raid +ext4 vs btrfs

hi,

would it make sense to test btrfs on disk device compared to a mdadm raid 1 on lvm2 partition. Most of us use raid or/and Lvm2 to provide raid and snapshotting to ext4 partition. So basicaly it would be fair to test a settup with

Disk
Soft RAID 1
Lvm2
Ext4

vs

Disk
Btrfs in raid1 like setup


Both present similar features so this could be very interesting to test.


regards,
Ghislain.