Announcement

Collapse
No announcement yet.

A Nicely-Built 40-Core Raspberry Pi Cluster

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by caligula View Post
    Man, WTF. Odroid U3 community edition is $59 and has quad 1.7 GHz Exynos cores and 2 GB of RAM ==> $14.75 per core
    Raspberry => $35 per core

    And you need 75% fewer SD cards, cables, switch ports.

    Guess which one is faster?
    So, there are a few things about that:
    • If performance/cost had been my basis for comparison, I would have put together an i7-based system with one or more fast video cards. It would have cost less and performed a lot better. A couple people would have congratulated me on building a nice gaming rig. You never would have heard about it.
    • Even though the work on this build didn't get serious until late last year, I was buying my first RPis in June 2012. It would not have been possible to choose the Odroid U3 for anything at that time. (As far as I know, it was only just released about a month ago.)
    • I wanted 32+ nodes in this cluster.
    • "And the maximum order quantity is limited to one unit for one person." - I could have gotten around that, but it would have been a pain and there probably would have been a lot of shipping to pay out.


    But if you want to build a cluster of Odroid U3s, I encourage you to do so. It sounds like it would be pretty sweet.

    Comment


    • #12
      Originally posted by DaveG View Post
      So, there are a few things about that:
      • If performance/cost had been my basis for comparison, I would have put together an i7-based system with one or more fast video cards. It would have cost less and performed a lot better. A couple people would have congratulated me on building a nice gaming rig. You never would have heard about it.
      • Even though the work on this build didn't get serious until late last year, I was buying my first RPis in June 2012. It would not have been possible to choose the Odroid U3 for anything at that time. (As far as I know, it was only just released about a month ago.)
      • I wanted 32+ nodes in this cluster.
      • "And the maximum order quantity is limited to one unit for one person." - I could have gotten around that, but it would have been a pain and there probably would have been a lot of shipping to pay out.


      But if you want to build a cluster of Odroid U3s, I encourage you to do so. It sounds like it would be pretty sweet.
      Whatever. If you don't count cores as nodes. If you use MPI, it also works with SMP systems as you might know. You could also have ordered two Parallela boards. The 16 core versions were already shipped to early backers. RPi is such a poor choice not because of slow CPU but because everything in RPi is a bottleneck. The network interface is slow, SD card reader is slow and buggy, USB power issues. On top of that the CPU is a major pain in the neck if you need to compile your software and you often do. The hardware is so slow you can't even use your cluster to provide any speedup when dist-compiling your software. If you had bought a tiny bit faster ARM boards, you could speed up with distcc.

      Comment


      • #13
        Originally posted by caligula View Post
        Man, WTF. Odroid U3 community edition is $59 and has quad 1.7 GHz Exynos cores and 2 GB of RAM ==> $14.75 per core
        Raspberry => $35 per core

        And you need 75% fewer SD cards, cables, switch ports.

        Guess which one is faster?
        Last time I checked, you're limited to 1 odroid U3 per person. That would be kind of hard to do a cluster that way. Otherwise, I'd agree - a single U3 is probably as good as at least 8 RPis and only costs twice as much. I believe there's a way to control the "heartbeat" LED on it too.

        As for my suggestion about the MK808 and using USB ethernet, keep in mind that 99% of all ARM platforms with built-in ethernet have a USB based ethernet. Unless you don't intend to use the RPi's serial ports, the ethernet jack on it I believe is on the same hub as the other USB ports, which could potentially slow it down further. Also, I wasn't suggesting to use the built-in wifi, because that is a terrible idea to use.

        Comment


        • #14
          Originally posted by schmidtbag View Post
          Last time I checked, you're limited to 1 odroid U3 per person. That would be kind of hard to do a cluster that way. Otherwise, I'd agree - a single U3 is probably as good as at least 8 RPis and only costs twice as much. I believe there's a way to control the "heartbeat" LED on it too.

          As for my suggestion about the MK808 and using USB ethernet, keep in mind that 99% of all ARM platforms with built-in ethernet have a USB based ethernet. Unless you don't intend to use the RPi's serial ports, the ethernet jack on it I believe is on the same hub as the other USB ports, which could potentially slow it down further. Also, I wasn't suggesting to use the built-in wifi, because that is a terrible idea to use.
          You're overly generous towards RPi. U3 has next gen CPU architecture compared to RPi. This already makes it twice as fast in some benchmarks. The clock rate is over twice as big. Then, 4 times as many cores. Guess how much faster interconnect is between CPU cores than USB powered ethernet? If the interconnect was 10 Gbps for RPi, U3 would be about as fast as 16 RPis. Since it's not and there's congestion and overhead from MPI or other distributed protocols, I'd say U3 is equivalent to 20-24 RPis.

          Comment


          • #15
            For the price of a 40 core PI cluster you could easyly buy 2x Intel Xeon E5 with 10 cores, which means 20 cores in one box with over 2 ghz each. Maybe for massive parallel apps a Xeon Phi with 61 cores active would be interesting as well. Using clusters of extra slow ARM v6 system leads to nothing, it is just crazy spent money. I have nothing against ARM in general and a PI can be used as low cost (VDR) server or XBMC client or for lots of other fun projects but it was never designed for HPC usage.

            Comment


            • #16
              Originally posted by Nobu View Post
              Well, sure, but a 486 won't be able to do a whole lot of concurrent operations. That's like comparing a CPU to a GPU--they're different beasts, with different purposes.

              That said, he was doing it sort-of for fun, but it started out as a thesis project. According to his blog post, he's going to use it to test distributed software.
              486 is perfectly capable of process concurrency.
              But my point was partially in jest anyway, for reasons as pointed out by the builder of the system. For raw power, it just isn't a huge bang for the buck, yet it is still a fun project.
              Last edited by droidhacker; 02-17-2014, 12:28 PM.

              Comment


              • #17
                I think the project is great.

                Sure, RPi is a poor choice, but let's consider it's a test build, made mostly out of fun, in which the main goal is the number of NODES (not cores!) and that computational power is not the ultimate purpose (otherwise the author is right:go for a i7 + GPU)

                The DIY acrilic/hardware case is really badass AND good looking

                My major concern for using such a rig for testing massively parallel software is the tremendous bottleneck caused by the internal ethernet network.

                In any case, congrats on your build; and I'm sure those 3 grands gave you not only lots of fun, but a good "visibility" and a nice story to add to you resum?.
                Netrunner Linux - Rolling Release ; Nexus 5 ROM Chroma 5.1 ; NAS 6TB on FreeNAS

                Comment


                • #18
                  Agreed. This is a very slick build, both in terms of "contents" and execution.

                  I find building a modern PC kind of depressing -- you put a few components together and everything just works. Not like the old mainframes... I miss the miles of beautifully managed cabling, aisles of components, and the big panels of blinky lights showing activity in each page of memory
                  Last edited by bridgman; 02-17-2014, 01:48 PM.

                  Comment


                  • #19
                    Originally posted by bridgman View Post
                    Agreed. This is a very slick build, both in terms of "contents" and execution.

                    I find building a modern PC kind of depressing -- you put a few components together and everything just works. Not like the old mainframes... I miss the miles of beautifully managed cabling, aisles of components, and the big panels of blinky lights showing activity in each page of memory
                    I agree - as a PC enthusiast, its somewhat sad that whenever I intend to build a new rig, it's probably going to be no bigger than a micro ATX with 1 stick of RAM, a cpu, a single SSD, and a single GPU. There won't be a need to put in anything else, but the only advantage to the computer vs an OEM brand is the ability to overclock. Even overclocking has become a pretty boring market lately. Anyone can breach 4GHz these days without even trying, and there's a good chance you don't even need the extra speed to begin with.

                    I'm thinking 10 years from now, all PCs will basically be the same platform, they'll all be pocket sized, and the only large computers will be either mainframes to handle unfathomable amounts of data, or quantum computers (which will basically do the same thing). Technology is going to look boring pretty soon.

                    Comment


                    • #20
                      You can always go retro, and build a Beowulf cluster out of 2010 parts in 2050.

                      Of course there's new areas of exciting tech too. For example 3d printing, the moment you can print metal items affordably, I'm so in
                      The current plastic items aren't that useful, the print area is small, and the quality bad. Some of the new resin-based printers are starting to have acceptable quality vs the filament-based ones though.

                      But once you can print big metal items with good quality, the things you can do will skyrocket. Yes I'll download a car.

                      Comment

                      Working...
                      X