Announcement

Collapse
No announcement yet.

Linux Sees A Slew Of Point Releases Due To That Nasty IBM POWER9 Vulnerability

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    I haven't seen any lockups with v5.9.9, but I woke up this morning to find both of my P9 boxes having input/output errors on their XFS filesytems. One box only saw it on its /home filesystem so I managed to get this out of /var/log/syslog (which is also on XFS):

    [160843.255360] XFS (bcache0): xfs_do_force_shutdown(0x8) called from line 461 of file fs/xfs/libxfs/xfs_defer.c. Return address = 0000000065546a41

    There was a stacktrace in dmesg, but I was too busy trying to recover the box and didn't think to save it, but I remember seeing something like detecting in memory corruption.

    The other box (which isn't bcache at all) is using a single root XFS filesystem for everything, and I wasn't able to get anything out of that.

    When initially trying to bring up both boxes, I let them both boot to v5.9.9 and and the affected filesystems refused to mount:
    Code:
    < bno + len at line 578 of file fs/xfs/libxfs/xfs_rmap.c. Caller xfs_rmap_unmap+0x79c/0xaa0
    [ 7.763152] CPU: 37 PID: 3069 Comm: mount Not tainted 5.9.9-64k-pages #182
    [ 7.763174] Call Trace:
    [ 7.763182] [c000002f9ac1f4c0] [c00000000082c1e0] dump_stack+0xc4/0x114 (unreliable)
    [ 7.763200] [c000002f9ac1f500] [c00000000069bfb4] xfs_corruption_error+0xf4/0x100
    [ 7.763224] [c000002f9ac1f5a0] [c00000000067b654] xfs_rmap_unmap+0x774/0xaa0
    [ 7.763239] [c000002f9ac1f6b0] [c000000000680b5c] xfs_rmap_finish_one+0x32c/0x3a0
    [ 7.763274] [c000002f9ac1f7c0] [c0000000006e110c] xfs_rui_item_recover+0x27c/0x380
    [ 7.763308] [c000002f9ac1f890] [c0000000006e7054] xlog_recover_process_intents.isra.28+0x204/0x320
    [ 7.763334] [c000002f9ac1f910] [c0000000006e7aa0] xlog_recover_finish+0x40/0x120
    [ 7.763350] [c000002f9ac1f980] [c0000000006d31dc] xfs_log_mount_finish+0x7c/0x170
    [ 7.763383] [c000002f9ac1f9c0] [c0000000006c00fc] xfs_mountfs+0x55c/0x9c0
    [ 7.763415] [c000002f9ac1fa70] [c0000000006c70e4] xfs_fc_fill_super+0x394/0x5e0
    [ 7.763449] [c000002f9ac1fb10] [c000000000469334] get_tree_bdev+0x234/0x380
    [ 7.763481] [c000002f9ac1fbb0] [c0000000006c6358] xfs_fc_get_tree+0x28/0x40
    [ 7.763504] [c000002f9ac1fbd0] [c000000000466bdc] vfs_get_tree+0x4c/0x160
    [ 7.763526] [c000002f9ac1fc50] [c0000000004a33fc] path_mount+0x4cc/0xd20
    [ 7.763549] [c000002f9ac1fd10] [c0000000004a3cd0] do_mount+0x80/0xd0
    [ 7.763562] [c000002f9ac1fd70] [c0000000004a42f8] sys_mount+0x158/0x180
    [ 7.763585] [c000002f9ac1fdc0] [c0000000000336d0] system_call_exception+0x160/0x280
    [ 7.763619] [c000002f9ac1fe20] [c00000000000d940] system_call_common+0xf0/0x27c
    [ 7.763668] XFS (bcache0): Corruption detected. Unmount and run xfs_repair
    [ 7.763705] XFS (bcache0): Internal error xfs_trans_cancel at line 954 of file fs/xfs/xfs_trans.c. Caller xfs_rui_item_recover+0x2d4/0x380
    Interestingly, when booting back to v5.9.8, both filesystems mounted without any issues. On one box, I umounted, and ran xfs_repair on one of the systems without a single complaint. My PC's running v5.9.9 and XFS are completely fine so far.

    Seems a number of XFS changes made it into v5.9.9 so maybe some of them aren't PPC friendly? Or maybe these changes are a red herring and some other changes relating to disk I/O were made that are incompatible with PowerPC?
    https://cdn.kernel.org/pub/linux/ker...hangeLog-5.9.9

    In any case, I don't see and XFS changes/fixes for v5.9.10, so I'd recommend staying away from both of these kernels for now if you're running POWER+XFS. I'm currently running v5.9.10 on my testing box to see if the issue reproduces within the next 24-48 hours. Is anyone else encountering filesystem corruption issues on POWER with v5.9.9+? If I reproduce this with v5.9.10, I'll file a kernel bug report, and the more info I have, the better.

    Annoyingly enough, v5.9.9 finally fixed a longstanding memory allocation issue I had where after 12-13 hours of uptime, one of my P9 boxes could no longer start new KVM instances.

    Comment


    • #12
      The problem isn't Intel or AMD, the problem is branch prediction exploits and you can't have branch prediction exploits if there's no branch prediction and you can't have branch prediction if you don't have cache, but there's one huge problem with having a cacheless system. a modern 6ghz CPU with cache disabled is just as fast as if you underclocked it to 50mhz.


      A problem I think could be reduced by having a RISC design and give it 3x IPC, hey Quake I ran in the Sega Saturn and that thing didn't have any cache and even if it used 100% of both CPUs, the CPUs were only clocked at 28.6mhz, so combined would be 57.2 Mhz, just over the speedlimit of a Cacheless System. so a 50mhz RISC would be like a 150mhz Pentium MMX. We could further optimize by using exotic semi-conductors that could reduce bus latency, Seymour Cray experimented in building a computer made out of GaNs and it was very coincidental that he died in a car accident before he could finish it. Anyway, I hear GaNs have a limitation of 50ghz and I wonder if the cacheless design scales, if so, you could scale up to 500mhz per core without cache and that would be like a Pentium III overclocked to 1.5ghz. Cache is also very die expensive because it's SRAM, it's 48 transistors per byte of SRAM though I hear what IBM has with their Mainframe CPUs, they use a DRAM cache so instead of having 6 transistors per bit, they hist have 1 transistor+1 capacitor, but let's imagine having really close to the CPU SRAM acting as a really fast RAM instead of cache. It's said the Apple M1 is 16B transistors and that's without 3D stacking, so let's imagine you can have one layer of something the same die size just for SRAM. 16000000000÷8÷6÷1024÷1024= 317MB of SRAM per layer and if you can get 8 layers of SRAM, that would be 2.5GB, still not a lot by RAM standards, but combined with latency improvements from exotic semi-conductors, that could be like an L4 Cache as fast as L1 is today and it could have more overhead for cache misfires if you insist on using cache and something like this would resurrect the amiga fast ram concept. But even if we were limited to something as fast as a 150Mhz x86 equivalent per core, we could have more optimized software and fixed function logic for computationally expensive stuff. We could have LUTs of ROM cache for Constants, (if we don't already) we could also have hardware made to accelerate codecs and encryption. Like imagine what it would be like if back in 1990, PC vendors got these chips that fell off the back of a UFO that accelerated DSPs, AES256, ZSTD, H266, xHE-AAC, Opus, Codec2 with the wavenet decoder, it would have been revolutionary for the time and something like mumble would have possible on dial-up and Fantasmagoria would have needed only 1 mini CD instead of 7 full sized and would have looked better and an album could have been stored on 5 floppies. There could also be better designed software optimized for a low frequency manycore system. Let's use games as an example, AI could be better on a low frequency manycore system where you could have each AI entity have it's own core and for draw calls, you could have one 50Mhz core could probably call 3k polygons for 60fps but one core could be a piece of the NPC Governor core like Exodia in Yugioh has one card for each limb and a card for the head, well, you could have one core for the NPC AI that would run something like HTN Planing (A more evolved AI than Fear's GOAP system) and that could tell body meta-caller that would tell the divisions of the body the xyz axis of the roll, pitch and yaw of each division and you can do this because cores are cheap when you have a cacheless RISC design. I would also suggest a sandboxed FPGA that also has a layer of fast SRAM embedded for custom algorithm acceleration and retro gaming because when you have a FPGA built into your computer, you don't need a N64 emulator and I would also suggest a sandboxed x86 legacy chip and I would be happy if it was fast as a Skulltraill with AVX2 functionality and an embedded iGPU that's driver compatible with the Radeon HD 7970, good enough for Windows 10 and backwards compatible with the Windows XP legacy.
      Last edited by commodore256; 22 November 2020, 01:03 PM.

      Comment


      • #13
        Originally posted by commodore256 View Post
        ... Seymour Cray experimented in building a computer made out of GaNs and it was very coincidental that he died in a car accident before he could finish it. Anyway, ...
        Well, thanks for the wall of text and the story of the coincidental car crash.

        The problem with the caches is currently mitigated by simply flushing them in between task switches. There is however no need to get rid of caches altogether. They still work very well.

        We then have other methods in place to make these attacks difficult. While caches can be used to snoop out memory, do we also randomise the layout of address spaces and other sensitive data.

        But just getting to the point where an attacker can execute malicious code on a machine (to exploit the caches) means in most cases that it's already too late anyway. At this point will it be easier for an attacker to exploit known bugs of the OS than trying to snoop for a new hole. An attacker might also not always need root access (i.e. to mine coins or start a DDoS against another system). In any case do you want to deny attackers access to your system much sooner.

        In short, the problem is real but also negligible for many, so you don't need to give up on caches just yet.

        Comment


        • #14
          Originally posted by Vistaus View Post

          I keep saying the same, but diehard AMD fans keep telling me AMD is safe...
          It's much safer so far and it's a fact. Furthermore, ARM CPUs are far more popular (and far less vulnerable than Intel), so this old fairy tale about AMD not being popular enough is getting boring.

          Comment


          • #15
            Originally posted by commodore256 View Post
            could be reduced by having a RISC design [...] give it 3x IPC [...] a 50mhz RISC would be like a 150mhz Pentium MMX [...] could further optimize by using exotic semi-conductors [...] you could scale up to 500mhz per core without cache and that would be like a Pentium III overclocked to 1.5ghz [...] I hear what IBM has with their Mainframe CPUs, they use a DRAM cache so instead of having 6 transistors per bit [...]
            You talk the talk, but do you walk the walk?

            Comment


            • #16
              Originally posted by uxmkt View Post
              You talk the talk, but do you walk the walk?
              It's all theoretical Computer Engineering, not applied Computer Engineering. (yet) But even if it stays at 50mhz RISC, you could do a lot with a manycore architecture with a lot of fixed function logic and optimized code. Oh, and here's the wikipedia page on their z15 Mainframe CPU, it uses embedded DRAM Cache.

              https://en.wikipedia.org/wiki/IBM_z15_(microprocessor)

              https://www.youtube.com/watch?v=0rNEtAz3wJQ

              Comment


              • #17
                Originally posted by Ipkh View Post

                AMD remains hampered by relying on 3rd party fans after the failure of Global Foundries. There are only 3 companies with interest in leading edge foundry operations and Intel is one of them. I'm not truly sure Samsung is really trying to keep up with the bleeding edge anymore. So really, TSMC and Intel are the only top tier fabs and only TSMC sells to outsiders. The huge drag on top tier fab capacity remains ASML and their EUV machines. Until fab spacecexoabds dramatically AMD cannot really regain much market from Intel.
                AMD is not hampered by the separation of GloFlo. The market in fabrication is being broken up into boutique niches. AMD was just ahead of where it was going simply because they couldn't provide the constant need for capital to upgrade the nodes.

                Where a fab was completely vertically integrated at one time, now there are firms that are taking on some of the activities of that integration. Recently when IBM sold their fabs to GloFlo, one of the first things that happened was GloFlo sold off the taping and wafer production to ON Technology. And then announced they would not invest in anymore advanced nodes themselves.

                The dis-integration of the chip production process is what is going to hurt Intel (not AMD) in the long run, because they have always preferred to stay completely vertically integrated.

                With the marketplace now seeking lower cost suppliers of pre-fabrication services not tied to a particular fab supplier, this will put production output directly in line with demand and hopefully lower costs.




                Comment


                • #18
                  Originally posted by Volta View Post

                  It's much safer so far and it's a fact..
                  Just to be sure, you are aware of the AMD PSP, correct? And the dangers of applying the same master cryptographic key to every CPU sold in a product line?

                  All that said, thus far the performance impacts for POWER9 are negligible for real world applications: https://twitter.com/carlosedp/status...276081671?s=20

                  Comment


                  • #19
                    Originally posted by madscientist159 View Post
                    thus far the performance impacts for POWER9 are negligible for real world applications: https://twitter.com/carlosedp/status...276081671?s=20
                    That's great news! Hope it keeps up the zero impact across the majority of benchmarks.

                    Comment


                    • #20
                      Originally posted by hiryu View Post
                      In any case, I don't see and XFS changes/fixes for v5.9.10, so I'd recommend staying away from both of these kernels for now if you're running POWER+XFS. I'm currently running v5.9.10 on my testing box to see if the issue reproduces within the next 24-48 hours. Is anyone else encountering filesystem corruption issues on POWER with v5.9.9+? If I reproduce this with v5.9.10, I'll file a kernel bug report, and the more info I have, the better.
                      To follow up with my claims here... So far, it seems like v5.9.10 is probably safe... And perhaps more interestingly, I definitely did experience the same issue with v5.9.9 on one of my amd64 machines after all.

                      Comment

                      Working...
                      X