Building A 96-Core Ubuntu ARM Solar-Powered Cluster

[SolarNet]

We ended up going with local SD card. We had some early success with

http://pandaboard.org/content/usb-downloader-and-usb-second-stage-bootloader-omap4

but then discovered that it wouldn't work with large usb hubs like

http://www.thinkgeek.com/product/eecf/?pfm=homepage_Featured_2_eecf

probably because you are transitting 2 USB chipsets on most of your pairwise client connections... we were going to revisit that issue some day because it would be neat if we could get USBNet completely working (from boot onwards)...

[ChuckAKnight]

Bootup

How did you bootup all 96 of the boards? Did you use a remote boot machanism like tftp?

[allaun]

I recommend turning on audio transcribing.

The video becomes hilarious. Some awesome quotes:
Benchmark certificates for justice
Film architecture is substantially different substantial is substantial
People disclousure experiment
It goes on. On a serious note, I'd suggest having captioning enabled for those that are hearing impaired.

[SolarNet]

Fair enough...

OK, we will put some of that comparison data together... probably over this coming weekend. We have tried a lot of experiments with outdoor supercomputing so we should be able to drum that data up.

We looked at the Variscite 4460 SOM a while back but we don't have any particular expertise with SODIMM. Are you only using that bus for power or is there a practical limitation to how many modules you can chain up that way?

As you probably know, Alex Ramirez (or just A-Ram in Barcelona) is espousing the Tegra board for his approach to hyperdense ARM servers. The Pandaboard is about .5U because of it's unique NIC arrangement, stacked over the USB, so you have to try out some exotic geometries to get better density. The MontBlanc board appears to be less than that... I guess these are the QSeven form factor? Check out from about 1:45 onwards at...

http://www.youtube.com/watch?v=_cZdTSOrq4w&feature=youtu.be

[mitcoes]

Budget and benchmark vs Intel and AMD x86

I would like to see a budget with total costs

And a benchmark vs a similar AMD and Intel budget machine

And another benchmark vs a similar CPU & GPU power - specs - x86_64 machine

This second one with prices, and power and of course the annual electricity bill for both systems and the number of solar panels needed for both systems.

As there are not ARM desktop array computers with similar CPU&GPU power - specs - machines - AT GENERAL STORES -, and having no need for funs, - ANNOYING NOISE - a project to make a desktop ARM computer with enought power to perform as actual AMD or Intel desktop computers and gaming computers - high end ones - it would be a great idea to make a how to - for human beings - to make a home one with panda boards, and in a near future ARM desktop clusters for home and gaming computers.

[popper]

thanks for the "

Kurt Keville of MIT and I talking about the ARM cluster:" clip it clears a lot up on the purpose of this

in case the other MIT guys want to try and make that micro ARM on SODIMM setup apparently they do also have a dual OMAP4460 1.5 GHz with 1Gigabit/s via on-module SMSC LAN7500 http://www.directinsight.co.uk/produ...omap-4460.html but no ARM quad versions off the shelf yet so far and that's a shame, so an I.mx6 quad 1.2 GHz 1Gigabit/s Ethernet although slower might still be the better option if they add that soon.

have you determined if your restricting throughput by the 10/100 Ethernet yet ? and did a small cluster (over 3 channels) running over wireless 11n ad hoc give better throughput ? i cant seem to find any numbers about real life wireless 11n throughput on these boards so that might be interesting to know

[SolarNet]

Additional details on Panda Cluster

We will try to get the specs up on the web. Meanwhile, here are the basics (I am one of the guys who worked on it).

We started with Chris Tyler's excellent HOWTO...

http://blog.chris.tylers.info/index.php?/categories/23-PandaBoard-Planet

and then bought Koutech 13-port USB hubs... we were chugging along pretty well in the 5v DC domain until we had to get a 48-port ethernet switch, which was a 3Com 2250 Plus (48 FastE and 2 GigE ports). Case was a Rubbermaid 9P90 Plaza Jr. (35 Gallon) Container which looks a lot like a Seahorse Power BigBelly.

Solar system was a "solar popcorn cart" from http://futuresolar.wordpress.com/ I think we will revisit this part of the equation; one of our guys has a cool MPPT controller (some pix of which used to be at http://solarwirelessmile.com/)... so next time we will try this without battery.

I'll get some more material together for a website and report back. We got good advice from some of the regular posters here. User drjo suggested we get rid of SD and use something else for the fs. We have tried booting with the usboot utility from Pandaboard.org with mixed results. Must not like the hubs, we think. Anyway, more to come.

[jimerickson]

excellent

i have been waiting for this. thank you michael!

[popper]

This solar panel shown in the pics wasn't built explictly for the cluster... They just happened to have the panel around plus a 400W inverter with battery... They also have a nice MIT solar concentrator, etc. The solar portion isn't part of the cluster build itself.

I'll be back out in Boston tomorrow to speak at BLU so might have another chance to play with the cluster some more, but right now there seems to be some performance issues in software.

what was the purpose of this 96 core build BTW ? for personal fun ,to prove the students can perform cobbling together off the shelf generic kit, or to perform a real test and write a paper etc

and when your there Michael please ask them why they didn't get MIT to buy in a load of cheap ARM cortex A9 on sodimm like these http://www.karo-electronics.com/tx53.html and have the Electrical Engineering and Computer Science guys knock up a 100 SODIMM slot carrier board interface to carry all the I/O signals to and from the TX module so as to get something with far lower power usage or just get the company to knock something simple and functional to cluster 10 SODIMM ARM cards with 4 SOC on each.

while the Karo electronics guys only show a set of slower speed Freescale i.MX535, A8 at 1 GHz,
im not sure so check but i think the Freescale i.MX6, 1.2 GHz A9 Quad (its also got 1gigabit Ethernet in soc)is pin compatible with all their other series so MIT could have karo or some other ARM on SODIMM vendors make a striped down PCB with all the ports and power passed on to the daughter board and a set (2 or 4 per card) of i.MX6 quads per SODIMM

http://www.karo-electronics.com/tx-standard.html

"

Concept

The TX embedded module integrates all the core components of a common PC and is mounted
onto an application specific carrier board. TX modules have a standardized form factor of 67,6mm x
26mm, have specified pinouts and provide the functional requirements for an embedded
application. These functions include, but are not limited to, graphics, network and multiple USB
ports. A single ruggedized SO-DIMM connector provides the carrier board interface to carry all the
I/O signals to and from the TX module. This SO-DIMM connector is a well known and proven high
speed signal interface connector that is commonly used for memory cards in notebooks.

Carrier board designers can utilize as little or as many of the I/O interfaces as deemed necessary.
The carrier board can therefore provide all the interface connectors required to attach the system
to the application specific peripherals. This versatility allows the designer to create a dense and
optimized package, which results in a more reliable product while simplifying system integration.
Most importantly, TX applications are scalable, which means once a product has been developed,
the product range can be diversified by using TX modules with a different performance class.
Simply unplug one module and replace it with another, no redesign is necessary."

[Michael]

I am mostly interested in the solar stuff. If they just got shelf bought components or if they did any type of customizations etc.

I believe you ll keep us updated.

This solar panel shown in the pics wasn't built explictly for the cluster... They just happened to have the panel around plus a 400W inverter with battery... They also have a nice MIT solar concentrator, etc. The solar portion isn't part of the cluster build itself.

I'll be back out in Boston tomorrow to speak at BLU so might have another chance to play with the cluster some more, but right now there seems to be some performance issues in software.