Announcement

Collapse
No announcement yet.

Building A 96-Core Ubuntu ARM Solar-Powered Cluster

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Building A 96-Core Ubuntu ARM Solar-Powered Cluster

    Phoronix: Building A 96-Core Ubuntu ARM Solar-Powered Cluster

    Last week I shared results from the Phoronix 12-core ARM Linux mini cluster that was constructed out of six PandaBoard ES development boards. Over the weekend, a 96-core ARM cluster succeeded this build. While packing nearly 100 cores and running Ubuntu Linux, the power consumption was just a bit more than 200 Watts. This array of nearly 100 processor cores was even powered up by a solar panel.

    http://www.phoronix.com/vr.php?view=17489

  • #2
    Michael details on the panel power supplies electrics etc will be published by MIT right??

    Comment


    • #3
      Originally posted by 89c51 View Post
      Michael details on the panel power supplies electrics etc will be published by MIT right??
      Maybe. It's pretty simple. They used a bunch of 13-port USB hubs. They could run about 8 PandaBoards off a 13-port USB hub with a 4Amp AC power adapter for each hub (0.5 Amps per PandaBoard). The PandaBoard non-ES can run off USB power by default, the ES is supposed to be able too, but evidently it's flakey whether or not it works.
      Michael Larabel
      http://www.michaellarabel.com/

      Comment


      • #4
        Power consumption is high in idle.

        3W in Idle for an omap4430, is a bit huge
        I know some smartphones powered by omap4430 that use only 5mA in idle on 4.7V battery, so about 24mW.

        normally the core can be switched OFF at runtime, and most peripheral too.
        So it seems Ubuntu can't do it (maybe due to the ethernet port ?).

        Comment


        • #5
          Originally posted by Michael View Post
          Maybe. It's pretty simple. They used a bunch of 13-port USB hubs. They could run about 8 PandaBoards off a 13-port USB hub with a 4Amp AC power adapter for each hub (0.5 Amps per PandaBoard). The PandaBoard non-ES can run off USB power by default, the ES is supposed to be able too, but evidently it's flakey whether or not it works.
          I am mostly interested in the solar stuff. If they just got shelf bought components or if they did any type of customizations etc.

          I believe you ll keep us updated.

          Comment


          • #6
            Originally posted by 89c51 View Post
            I am mostly interested in the solar stuff. If they just got shelf bought components or if they did any type of customizations etc.

            I believe you ll keep us updated.
            This solar panel shown in the pics wasn't built explictly for the cluster... They just happened to have the panel around plus a 400W inverter with battery... They also have a nice MIT solar concentrator, etc. The solar portion isn't part of the cluster build itself.

            I'll be back out in Boston tomorrow to speak at BLU so might have another chance to play with the cluster some more, but right now there seems to be some performance issues in software.
            Michael Larabel
            http://www.michaellarabel.com/

            Comment


            • #7
              Originally posted by Michael View Post
              This solar panel shown in the pics wasn't built explictly for the cluster... They just happened to have the panel around plus a 400W inverter with battery... They also have a nice MIT solar concentrator, etc. The solar portion isn't part of the cluster build itself.

              I'll be back out in Boston tomorrow to speak at BLU so might have another chance to play with the cluster some more, but right now there seems to be some performance issues in software.
              what was the purpose of this 96 core build BTW ? for personal fun ,to prove the students can perform cobbling together off the shelf generic kit, or to perform a real test and write a paper etc

              and when your there Michael please ask them why they didn't get MIT to buy in a load of cheap ARM cortex A9 on sodimm like these http://www.karo-electronics.com/tx53.html and have the Electrical Engineering and Computer Science guys knock up a 100 SODIMM slot carrier board interface to carry all the I/O signals to and from the TX module so as to get something with far lower power usage or just get the company to knock something simple and functional to cluster 10 SODIMM ARM cards with 4 SOC on each.

              while the Karo electronics guys only show a set of slower speed Freescale i.MX535, A8 at 1 GHz,
              im not sure so check but i think the Freescale i.MX6, 1.2 GHz A9 Quad (its also got 1gigabit Ethernet in soc)is pin compatible with all their other series so MIT could have karo or some other ARM on SODIMM vendors make a striped down PCB with all the ports and power passed on to the daughter board and a set (2 or 4 per card) of i.MX6 quads per SODIMM

              http://www.karo-electronics.com/tx-standard.html
              "
              Concept

              The TX embedded module integrates all the core components of a common PC and is mounted
              onto an application specific carrier board. TX modules have a standardized form factor of 67,6mm x
              26mm, have specified pinouts and provide the functional requirements for an embedded
              application. These functions include, but are not limited to, graphics, network and multiple USB
              ports. A single ruggedized SO-DIMM connector provides the carrier board interface to carry all the
              I/O signals to and from the TX module. This SO-DIMM connector is a well known and proven high
              speed signal interface connector that is commonly used for memory cards in notebooks.

              Carrier board designers can utilize as little or as many of the I/O interfaces as deemed necessary.
              The carrier board can therefore provide all the interface connectors required to attach the system
              to the application specific peripherals. This versatility allows the designer to create a dense and
              optimized package, which results in a more reliable product while simplifying system integration.
              Most importantly, TX applications are scalable, which means once a product has been developed,
              the product range can be diversified by using TX modules with a different performance class.
              Simply unplug one module and replace it with another, no redesign is necessary."
              Last edited by popper; 06-19-2012, 07:24 PM.

              Comment


              • #8
                excellent

                i have been waiting for this. thank you michael!

                Comment


                • #9
                  Additional details on Panda Cluster

                  We will try to get the specs up on the web. Meanwhile, here are the basics (I am one of the guys who worked on it).

                  We started with Chris Tyler's excellent HOWTO...
                  http://blog.chris.tylers.info/index....daBoard-Planet

                  and then bought Koutech 13-port USB hubs... we were chugging along pretty well in the 5v DC domain until we had to get a 48-port ethernet switch, which was a 3Com 2250 Plus (48 FastE and 2 GigE ports). Case was a Rubbermaid 9P90 Plaza Jr. (35 Gallon) Container which looks a lot like a Seahorse Power BigBelly.

                  Solar system was a "solar popcorn cart" from http://futuresolar.wordpress.com/ I think we will revisit this part of the equation; one of our guys has a cool MPPT controller (some pix of which used to be at http://solarwirelessmile.com/)... so next time we will try this without battery.

                  I'll get some more material together for a website and report back. We got good advice from some of the regular posters here. User drjo suggested we get rid of SD and use something else for the fs. We have tried booting with the usboot utility from Pandaboard.org with mixed results. Must not like the hubs, we think. Anyway, more to come.

                  Comment


                  • #10
                    thanks for the "
                    Kurt Keville of MIT and I talking about the ARM cluster:" clip it clears a lot up on the purpose of this


                    in case the other MIT guys want to try and make that micro ARM on SODIMM setup apparently they do also have a dual OMAP4460 1.5 GHz with 1Gigabit/s via on-module SMSC LAN7500 http://www.directinsight.co.uk/produ...omap-4460.html but no ARM quad versions off the shelf yet so far and that's a shame, so an I.mx6 quad 1.2 GHz 1Gigabit/s Ethernet although slower might still be the better option if they add that soon.

                    have you determined if your restricting throughput by the 10/100 Ethernet yet ? and did a small cluster (over 3 channels) running over wireless 11n ad hoc give better throughput ? i cant seem to find any numbers about real life wireless 11n throughput on these boards so that might be interesting to know
                    Last edited by popper; 06-21-2012, 07:22 PM.

                    Comment


                    • #11
                      Budget and benchmark vs Intel and AMD x86

                      I would like to see a budget with total costs

                      And a benchmark vs a similar AMD and Intel budget machine

                      And another benchmark vs a similar CPU & GPU power - specs - x86_64 machine

                      This second one with prices, and power and of course the annual electricity bill for both systems and the number of solar panels needed for both systems.

                      As there are not ARM desktop array computers with similar CPU&GPU power - specs - machines - AT GENERAL STORES -, and having no need for funs, - ANNOYING NOISE - a project to make a desktop ARM computer with enought power to perform as actual AMD or Intel desktop computers and gaming computers - high end ones - it would be a great idea to make a how to - for human beings - to make a home one with panda boards, and in a near future ARM desktop clusters for home and gaming computers.

                      Comment


                      • #12
                        Fair enough...

                        OK, we will put some of that comparison data together... probably over this coming weekend. We have tried a lot of experiments with outdoor supercomputing so we should be able to drum that data up.

                        We looked at the Variscite 4460 SOM a while back but we don't have any particular expertise with SODIMM. Are you only using that bus for power or is there a practical limitation to how many modules you can chain up that way?

                        As you probably know, Alex Ramirez (or just A-Ram in Barcelona) is espousing the Tegra board for his approach to hyperdense ARM servers. The Pandaboard is about .5U because of it's unique NIC arrangement, stacked over the USB, so you have to try out some exotic geometries to get better density. The MontBlanc board appears to be less than that... I guess these are the QSeven form factor? Check out from about 1:45 onwards at...
                        http://www.youtube.com/watch?v=_cZdT...ature=youtu.be

                        Comment


                        • #13
                          I recommend turning on audio transcribing.

                          The video becomes hilarious. Some awesome quotes:
                          Benchmark certificates for justice
                          Film architecture is substantially different substantial is substantial
                          People disclousure experiment
                          It goes on. On a serious note, I'd suggest having captioning enabled for those that are hearing impaired.

                          Comment


                          • #14
                            Bootup

                            How did you bootup all 96 of the boards? Did you use a remote boot machanism like tftp?

                            Comment


                            • #15
                              We ended up going with local SD card. We had some early success with
                              http://pandaboard.org/content/usb-do...otloader-omap4
                              but then discovered that it wouldn't work with large usb hubs like
                              http://www.thinkgeek.com/product/eec...eatured_2_eecf
                              probably because you are transitting 2 USB chipsets on most of your pairwise client connections... we were going to revisit that issue some day because it would be neat if we could get USBNet completely working (from boot onwards)...

                              Comment

                              Working...
                              X