Announcement

Collapse
No announcement yet.

EXT4 Data Corruption Bug Hits Stable Linux Kernels

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Originally posted by Pallidus View Post
    LOL is on you because you could be running stable 3.6.1 and be unnafected as well.


    PROTIP wait for the kernels to mature, even the stable ones, for at least 15 days before upgrading them PROTIP


    "Still, the commit in question *does* change things, and so it's still the most likely culprit."


    name and shame plox
    I only use RC/Beta/Alpha kernels for years and most of the time Alpfa/Beta/Git Readeon driver as well.

    stable releases are for pussy’s.

    Comment


    • #32
      Originally posted by necro-lover View Post
      stable releases are for pussy’s.
      Stable releases are for people doing serious work who need their systems to function without chasing down kernel bugs all the time. I spent too much time on this as it is (though my employer has a direct interest in the stability of the Linux kernel, so nobody was too unhappy).

      I must say I'm very happy with responsiveness here: I first saw fs corruption on Monday, reported it on Tuesday after figuring out that it was definitely 3.6.3 at fault and thus not an already-fixed bug in an old stable kernel, and had a candidate patch from Ted within a few hours, even though I'd dropped this on him without warning and with so little info that he had to dig through every ext*-affecting patch between 3.6.1--3.6.3. I'm sure I couldn't respond to a bug described that vaguely anywhere near that fast. As ever, Ted provides the rest of us with something to aspire to!

      Comment


      • #33
        Originally posted by enrico.tagliavini View Post
        Feel free to help.
        Why do *I* have to "feel free" when RedHat is paying its testers perfectly good money?

        If you think you can read and understand in every detail hundred thousand lines of code you can safely replace Linus.
        Now you say one has to be a gnarly kernel hacker to help! Which is it?

        Software has bugs. It is simply impossible to dodge them all. Just think about the notorious random number generator in debian some stable release ago....
        Yes indeed I think all the time about the lack of even the most rudimentary sorts of regression testing.

        Just thank you the openness of linux, will hit only a very small fraction of linux users and most likely geeks and contributors
        Oh you mean only the people who are using the latest releases of the two most mainstream distributions?

        Comment


        • #34
          That's why kernel shoud have automatic tests. Code review is important but it's not a substitute to a good test coverage.

          Comment


          • #35
            Again, you can't spot such a bug with automatic tests.
            ## VGA ##
            AMD: X1950XTX, HD3870, HD5870
            Intel: GMA45, HD3000 (Core i5 2500K)

            Comment


            • #36
              Originally posted by tehehe View Post
              That's why kernel shoud have automatic tests. Code review is important but it's not a substitute to a good test coverage.
              And how do you test for errors you can't reproduce?

              Comment


              • #37
                Now I am confused...

                [hamish@Griffindor ~]$ yum info kernel
                Loaded plugins: langpacks, presto, refresh-packagekit
                Available Packages
                Name : kernel
                Arch : i686
                Version : 3.6.2
                Release : 1.fc16
                Size : 26 M
                Repo : updates
                Summary : The Linux kernel
                URL : http://www.kernel.org/
                License : GPLv2
                Description : The kernel package contains the Linux kernel (vmlinuz), the core
                : of any Linux operating system. The kernel handles the basic
                : functions of the operating system: memory allocation, process
                : allocation, device input and output, etc.

                [hamish@Griffindor ~]$

                [root@Griffindor ~]# yum update
                Loaded plugins: langpacks, presto, refresh-packagekit
                fedora-awesome | 2.8 kB 00:00
                fedora-chromium-stable | 3.4 kB 00:00
                rpmfusion-free-updates | 3.3 kB 00:00
                rpmfusion-nonfree-updates | 3.3 kB 00:00
                updates/metalink | 16 kB 00:00
                No Packages marked for Update
                [root@Griffindor ~]#

                [hamish@Griffindor ~]$ uname -r
                3.4.11-1.fc16.i686.PAE
                [hamish@Griffindor ~]$

                I guess this is good, but...

                Comment


                • #38
                  Originally posted by PuckPoltergeist View Post
                  And how do you test for errors you can't reproduce?
                  Quite. My latest tests suggest that you have to reboot *while a umount is in progress* for this to go wrong -- and that this affects Linux 3.6.1 and quite possibly many earlier versions (untested as yet), though the dangerous race window is much narrower in kernels before 3.6.2 or 3.6.3 and you pretty much have to do the umount and then the reboot -f as the very next command to make it go wrong. It is not plausible that anyone would have thought of testing *that* before I ran into it. But my home server is a test platform that does just that!

                  This is, to be honest, a somewhat insane thing to do, even though I need to do it in order to reboot reliably due to nested NFS and non-NFS mounts, not all of which may be reachable at umount time. I'm not entirely convinced this is even a bug, though I hope it's a bug because I'm sick of seeing my filesystems corrupted!

                  It certainly explains why, myself apart, only people using ext4 on removable devices have seen it so far (though anyone making heavy use of umount -l in any context would probably see it soon enough).

                  Comment


                  • #39
                    I have a Google+ post where I've posted my latest updates:

                    https://plus.google.com/117091380454...ts/Wcc5tMiCgq7

                    I will note that before I send any pull request to Linus, I have run a very extensive set of file system regression tests, using the standard xfstests suite of tests (originally developed by SGI to test xfs, and now used by all of the major file system authors). So for example, my development laptop, which I am currently using to post this note, is currently running v3.6.3 with the ext4 patches which I have pushed to Linus for the 3.7 kernel. Why am I willing to do this? Specifically because I've run a very large set of automated regression tests on a very regular basis, and certainly before pushing the latest set of patches to Linus. So for all of the kvetching about people not willing to run bleeding edge kernels, please remember that while it is no guarantee of 100% perfection, I and many other kernel developers *are* willing to eat our own dogfood.

                    Is there more testing that we could do? Yes, as a result of this fire drill, I will probably add some systematic power fail testing before I send a pull request to Linus. But please rest assured that we are already doing a lot of QA work as a regular part of the ext4 development process already.

                    Comment


                    • #40
                      Originally posted by NullNix View Post
                      Stable releases are for people doing serious work who need their systems to function without chasing down kernel bugs all the time. I spent too much time on this as it is (though my employer has a direct interest in the stability of the Linux kernel, so nobody was too unhappy).

                      I must say I'm very happy with responsiveness here: I first saw fs corruption on Monday, reported it on Tuesday after figuring out that it was definitely 3.6.3 at fault and thus not an already-fixed bug in an old stable kernel, and had a candidate patch from Ted within a few hours, even though I'd dropped this on him without warning and with so little info that he had to dig through every ext*-affecting patch between 3.6.1--3.6.3. I'm sure I couldn't respond to a bug described that vaguely anywhere near that fast. As ever, Ted provides the rest of us with something to aspire to!
                      be happy that people like me use the latest beta/alpha stuff to trash there system because I also report bugs if i find one

                      Comment


                      • #41
                        It isn't too hard to hit this on Fedora 17. Install clean system, do upgrades, reboot, you are running kernel 3.6.2 now, then install drivers, reboot, install anything and filesystem goes read-only, reboot, system won't boot, you need to fsck.

                        Comment


                        • #42
                          Originally posted by henrymiller View Post
                          It isn't too hard to hit this on Fedora 17. Install clean system, do upgrades, reboot, you are running kernel 3.6.2 now, then install drivers, reboot, install anything and filesystem goes read-only, reboot, system won't boot, you need to fsck.
                          Please send a detailed report to the linux-ext4 mailing list. Include the kernel version you are running, the kernel logs (from dmesg) that were printed or which you found in the logs, the output from e2fsck (which are hopefully saved in /var/log/fsck), and whether you can reliability reproduce it or how frequently it happens.

                          There are many potential causes of file system corruption, and sometimes they can have very similar symptoms. This is why I hate try to debug such problems on web forums or on Ubuntu's Launchpad. Someone sees a relatively vague description of the problem on the web forum, immediately jump to the conclusion that it must be the same thing that they saw, and pile on the web thread. Always send us detailed information; if it's redundant with what someone else has sent, that's actually *good*. That way we can confirm a common pattern. But please don't assume a pattern where one doesn't exist; humans tend to do this even when it's not justified.

                          Comment


                          • #43
                            Quality Assurance

                            Quality Assurance (Q.A.) should have found that problem before the merge with mainline... Oh wait a minute, this is that Cathedral and Bazaar methodology being shoved down the throats of everyone.

                            Seriously speaking, maybe a double boot should be added like a marksman's double tap to your testing repertoire.

                            Maybe consider freezing all changes to Ext4 and make a new Ext5 for new ideas.

                            Assume nobody runs on Battery Backup.

                            Assume people reboot because the power kepts going out: "On then Off then On then off and a couple more times is common."

                            Always plan for the worst case.

                            Nevermind...

                            Comment


                            • #44
                              Originally posted by tytso View Post
                              I have a Google+ post where I've posted my latest updates:

                              https://plus.google.com/117091380454...ts/Wcc5tMiCgq7

                              I will note that before I send any pull request to Linus, I have run a very extensive set of file system regression tests, using the standard xfstests suite of tests (originally developed by SGI to test xfs, and now used by all of the major file system authors). So for example, my development laptop, which I am currently using to post this note, is currently running v3.6.3 with the ext4 patches which I have pushed to Linus for the 3.7 kernel. Why am I willing to do this? Specifically because I've run a very large set of automated regression tests on a very regular basis, and certainly before pushing the latest set of patches to Linus. So for all of the kvetching about people not willing to run bleeding edge kernels, please remember that while it is no guarantee of 100% perfection, I and many other kernel developers *are* willing to eat our own dogfood.

                              Is there more testing that we could do? Yes, as a result of this fire drill, I will probably add some systematic power fail testing before I send a pull request to Linus. But please rest assured that we are already doing a lot of QA work as a regular part of the ext4 development process already.
                              Did anyone seriously think that you were not doing this? Anyway, even if your patches were perfect, someone else could make an "improvement" elsewhere in the kernel that will introduce a bug or trigger a latent race condition. If patches sent to Linus were bug free in all situations, we would not need kernel releases.

                              On the topic of systematic power fail testing, I suggest using kexec. It makes such testing easy.

                              By the way, what is your opinion of data integrity in ZFS? I have never seen you comment on it.
                              Last edited by ryao; 10-24-2012, 11:38 PM.

                              Comment


                              • #45
                                Originally posted by squirrl View Post
                                Quality Assurance (Q.A.) should have found that problem before the merge with mainline... Oh wait a minute, this is that Cathedral and Bazaar methodology being shoved down the throats of everyone.
                                Nothing is shoved down anybody's throat. You don't have to use the latest bleeding edge kernel. Feel free to use Debian Stable, or RHEL 6 if you like. Heck, use Windows 8 if it floats your boat.

                                Seriously speaking, maybe a double boot should be added like a marksman's double tap to your testing repertoire.
                                Double boot was a theory; it turned out not to be true (remember, when we do our bug hunting on an open mailing list, sometimes early theories are exposed which turns out not to be true; it's unfortunate when people think it's worthwhile to put those theories in news stories just to drive web hits, for advertising $$$, but that's just the way things roll; don't believe everything you see on the web).

                                In fact we regularly mount and unmount the file system many times during the regression tests which I run all the time (which would have been equivalent to a double boot).

                                Assume nobody runs on Battery Backup.
                                Actually using the nobarrier mount options (which can only be used if you are using hardware raid or an enterprise storage array) is actually pretty rare outside of people who should be using enterprise Linux distributions; and the enterprise Linux distributions do a lot of testiing.

                                Assume people reboot because the power kepts going out: "On then Off then On then off and a couple more times is common."
                                I dogfood changes before they get pushed out, and I certainly do things like forced poweroffs. (Usually due to X server or suspend/resume bugs, oh, well.) Yes, we should probably add systematic powerfail testing to me regular regression testing that gets done during the development cycle. There's always room for improvement.

                                Always plan for the worst case.
                                Absolutely. We certainly design with this in mind. There is an awful lot of paranoia built into ext4. In fact, the ext4 error messages that got tripped was specifically because of the paranoid double checks that are built into ext4. You can even add the mount option "block_validity" which will increase the checks, at the cost of adding more CPU overhead --- I run with this mount option when I run my regression tests, for example.

                                Maybe consider freezing all changes to Ext4 and make a new Ext5 for new ideas.
                                We have considered this. Right now new features get added under experimental feature flags or mount options. One of the users who ran into problems were using experimental new features that are not enabled by default. We can't stop users from trying out new features that aren't enabled by default, just as we can't stop them from deciding to use ext5 instead of ext4 on production servers. Things like metadata checksums are not enabled by default specifically because they aren't ready yet. Brave users who try them out are invaluable, and I am grateful to people who help us do our testing, since that's the only way we can shake out the last bugs that aren't found in developer environments or via regression tests. But you make your choices, and take your chances when you turn on such experimental features.

                                And there are some real costs with forking the code base and creating a potential new "ext5". We already have had problems where bugs are fixed in ext4, and they aren't propagated to ext3. Just today I found a minor bug which was fixed in ext3, but not in ext2. And sometimes bugs are fixed in extN, but someone forgets to forward port the bug fix to extN+1. If we were to add an "ext5" it would make this problem much worse, since it would triple the potential ways that bug fixes might fail to get propagated to where they are needed.

                                Speaking of bug fixes, you can't freeze all changes, because we are still finding bugs. Heck, as part of deploying ext4 is deployed on thousands and thousands of thousands of machines in Google data centers, we found a bug that only showed up because we had deployed ext4 in great numbers. When we found the bug fix, I checked and found that the exact same bug existed in ext3, where it had not been found despite ten years of testing in enterprise linux releases, by companies such as IBM and Red Hat. (It had probably triggered a couple of times, but it was so rare that testers probably chalked it up to random hardware failure or cosmic rays; it was only because I was correlating failure reports --- and most were caused by hardware failures, not by software bugs --- across a very large number of machines that I could discern the pattern and find this particular bug.)

                                The problem is that sometimes bug fixes introduce other bugs. In this particular case, it was a bug fix which as backported to a stable kernel which apparently made this failure mode happen more often. If you really mean "freeze all changes", as opposed to just being full of snark, then that would also mean not taking any bug fixes. And if you want to stay on an older version of Linux, feel free..... that's what people who are using RHEL 5, or RHEL 4, or even in some cases RHAS 2.1 have chosen.

                                Comment

                                Working...
                                X