Systemd 219 Released With A Huge Amount Of New Features

[Ericg]

Thanks, and sorry about complaining about systemd.

systemd is actually very nice.

I feel like an idiot now.

Hey, just a heads up. Current reliability winner for HDD's is HGST. Western Digital is their parent company but HGST has their own products so don't assume "Oh WD owns HGST, WD's the same." They're not. So if you DO plan on replacing the drive, see what HGST has to offer. Tend to be a little more expensive (like $10-20) but they have the best reliability right now.

[grndzro]

Thanks, and sorry about complaining about systemd.

systemd is actually very nice.

I feel like an idiot now.

We have all been idiots on here a time or 2.

[ihatemichael]

A small walkaround until you have a new drive: https://wiki.archlinux.org/index.php...lesystem_Check
This will make EXT4 aware of bad blocks and avoid using them. Next time, slap BTRFS on the drive, it will report immediately if there's a problem (saved me one time)

Can you please tell me how BTRFS would have reported the errors immediately? What is that it's so fundamentally different in BTRFS that it would have reported errors immediately?

Also, where would these errors show up when using BTRFS? dmesg? the journal?

[duby229]

This isn't what the 252 means, I promise. If those numbers read 0 (at or below the third of those numbers), the drive is considered to be in a failure state.

Those numbers are the drive's interpretation of that particular characteristic. There's a reason half of the other attributes also have a 252 there - that's just the highest number the drive considers using (253, 254, and 255 may have special meanings). It's not that 252 of each of those different events happened... that would be extraordinarily unlikely.

Some other drives limit it to a percent scale and you'll see a bunch of 100's on the stuff the drive considers to be in perfect condition.

Oh, Thanks for that correction. I guess, I've been looking at it wrong. I just checked on another drive I know has bad sectors and what you said is true for this drive too.

I guess I can learn something if I don't let my stubborness get in the way.

[BradN]

Oh, Thanks for that correction. I guess, I've been looking at it wrong. I just checked on another drive I know has bad sectors and what you said is true for this drive too.

I guess I can learn something if I don't let my stubborness get in the way.

Hey don't feel bad about it - when we started out, we threw out a few drives due to excessive bad sectors but it was actually the version of linux we were using screwing up and no longer transferring any data after one error. Felt pretty stupid about that one!

[Ericg]

Can you please tell me how BTRFS would have reported the errors immediately? What is that it's so fundamentally different in BTRFS that it would have reported errors immediately?

Also, where would these errors show up when using BTRFS? dmesg? the journal?

btrfs checksums all data on the system. Ars has a write up on BTRFS: http://arstechnica.com/information-t...n-filesystems/ where they do some forced corruption tests. The long story short is if one BIT is swapped in a file when btrfs goes and checksums the data again it will detect that it's different than last time it checksummed it, and it shouldn't be. It will then yell at you, I think in dmesg, that it found corruption and it will say whether it fixed it itself (by copying over a known good copy. Yay Copy-on-write) or whether it couldnt fix it and just is yelling at you.

[SystemCrasher]

I faced that problem once. Most file systems corrupt without warning but only BTRFS reported the csum errors.

One of reasons why extra checksum could be good idea. While modern drives supposed to either return error-free data or I/O error (and possibly to try to correct problem, e.g. by remapping faulty sector/block), it sometimes not a case. Then there is high-speed link and while USB and SATA have some error detection, it usually weak (it have to be very fast in first place) and chance that bogus data will go past these checks aren't zero. Then there could be some other errors in hardware, be it faulty RAM or misbehaving CPU. Checksumming in filesystem could detect many of these errors and would at least prompt for investigation to find faulty parts. Also RAID implemented in filesystem can be explicitly aware which copy of data is good, etc in such cases. So it sometimes could be worth to spent some CPU cycles on extra checks if you demand more reliability.

Even badblocks couldn't find this. After reflashing the USB firmware, everything is fine now.

It could be unobvious in many cases and badblocks should at least do read-write test (which would also cost you about 1 write cycle on whole flash device, some noticeable part of device lifetime). And in fact you should do something like this: place semi-random data on all device area and check if you can read whole sequence back correctly.

Badblocks has been created at ages of mechanical devices. Flash based devices work in really different ways and you can't even expect badblocks will check more or less all cells of memory. You see, there is translator/wear leveller running in background, it tosses blocks as it sees it fits, to make sure blocks undergo more or less same amount of write cycles. So when you do badblocks run, there is no real warranty it would touch and check each and every memory cell. On HDD you can usually expect this, but HDDs do not need wear leveling.

I can imagine firmware reflash fixed internal structures, possibly forcing controller to rebuild tables from scratch, do thorough badblocks checks and so on. But it's not something easy to do - its really vendor specific stuff.

[alaviss]

Can you please tell me how BTRFS would have reported the errors immediately? What is that it's so fundamentally different in BTRFS that it would have reported errors immediately?

Also, where would these errors show up when using BTRFS? dmesg? the journal?

It should be written in the BTRFS docs so go check it out. Basicly BTRFS do a csum for all blocks on the machine, if any blocks have mismatched csum, it will be mark as invaild, however, you can't seem to recover data from the invaild blocks. The errors will show up in dmesg (and of course, will be captured by the journal).

[alaviss]

It could be unobvious in many cases and badblocks should at least do read-write test (which would also cost you about 1 write cycle on whole flash device, some noticeable part of device lifetime). And in fact you should do something like this: place semi-random data on all device area and check if you can read whole sequence back correctly.

Badblocks has been created at ages of mechanical devices. Flash based devices work in really different ways and you can't even expect badblocks will check more or less all cells of memory. You see, there is translator/wear leveller running in background, it tosses blocks as it sees it fits, to make sure blocks undergo more or less same amount of write cycles. So when you do badblocks run, there is no real warranty it would touch and check each and every memory cell. On HDD you can usually expect this, but HDDs do not need wear leveling.

I can imagine firmware reflash fixed internal structures, possibly forcing controller to rebuild tables from scratch, do thorough badblocks checks and so on. But it's not something easy to do - its really vendor specific stuff.

Actually destructive badblocks read/writes 5 times with 5 different pattern without findings... Firmware Reflash fails because there was too many bad blocks, so I simply force it to allocate more good blocks as spare blocks (killed 1 GB of space but worth it, the flash drive was 32GB anyway)

[Pajn]

In any case, you can setup journald to forward logs to whatever logging daemon you prefer, wouldn't that fix your problem (as you don't want to use journald in any case)?

Would that mean that I can get it to write standard, human readable text files?

As an Ubuntu user I will soon use systemd and the binary log files is my biggest concern, if I can get that fixed I would be much happier.