infectedtech: The AAPEN Project

[[Infected]]

AAPEN er kort for Adaptive Array of Power Efficient Nodes, og er ett prosjekt jeg holder paa med paa Murdoch University, School of Information Technology, som befinner seg I Perth, paa vestkysten av Australia.

 

 

 

I god tradisjon tro tenkte jeg det var passende aa poste build-loggen av prosjektet her

 

Many great components, an awesome primary node make

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL.

Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.

 

Kommentartraaden finner du her.

 

 

MvH,

[infected]

Endret 15. september 2011 av [Infected]

[[Infected]]

For enclosure, the Antec 4U22ATX has been chosen. This is a cheap yet solid rack mount case, with a reasonable price point, not to mention easily available without having to ship from the other coast.

 

 

 

Unboxing the Antec 4U22ATX, the 4 Rack Unit (RU) rack mountable housing destined for the primary node.

 

The Antec in all its glory..

 

The insides of the case.

 

Antec has not skimped on the fans, Yate Loon make some great fans.

 

Mainboard

 

Gigabyte 890FXA-UD7

 

Unboxing..

 

As it turns out, the 890FXA-UD7 is longer than standard full size ATX (its "XL-ATX" apparently, whatever that is), what a bummer!

 

The board has to be exchanged for a more 'standard' ATX friendly one..

 

 

 

 

MvH

[infected]

Endret 26. desember 2010 av [Infected]

[[Infected]]

Denne gangen er det lagring og komponenter til primaernoden som staar for tur.

 

 

 

The drive cage.

 

Samsung TruDirect DVD writer.

 

Quad 2.5" SATA drive backplane.

 

2x PQI 64GB SandForce powered SSD's, one 32GB A-RAM SSD, and one Seagate 320GB XT hybrid drive..

The PQI's and A-RAM are all MLC's, not to mention the PQI's come with 64MB dram cache. The XT has a 4GB SLC read cache, and a nifty 32MB dram cache. This should make for some interesting comparisons

 

The A-RAM getting comfy with the drive bay.

 

The MLC SSD trifecta.

 

Each drives sits happily in its respective cage

 

The 2nd drive cage will soon be filled with 7.2K rpm 500GB drives to be used for general purpose storage.

 

The 1040SAS Quartet Mobile Backplane, and its assorted cables..

 

Backplanes and DVD writer mounted.. FDD is missing for now.. damn, where did that black floppy drive go?

 

A rather flashy photograph of how the drive bay looks..

 

A nice pic of the rear end of the cage, too bad my camera handles flash so poorly.

 

The ever so classic modular bay promotional picture!

 

The AMD X6 1055T six-core processor.

 

16 Gigabytes of DDR3 1333MHz memory, in a 4x4GB configuration.

 

CUDA Accelerator

 

The EVGA GTS450 with 1GB of memory selected for this build.

 

EVGA GTS450, the GF106 should clock in at about 600 gigaflops.

 

 

 

More images to come

 

MvH

[infected]

Endret 26. desember 2010 av [Infected]

[[Infected]]

The New Mainboard

 

 

 

Thanks to the great service at http://www.msy.com.au/ replacing the UD7 with the more ATX friendly UD5 was no problem

 

Nothing much included, but a few SATA and IDE/floppy cables.

 

The 890FXA-UD5 in all its glory

 

The Powerplant

As power plant, the HX650 from Corsair has been selected.

 

The contents of the box. This power supply is 95% modular, leaving only the 24pin ATX and the ATX12V connector.

 

The ratings of the Corsair CMPSU-650HX

 

The CPU Heatsink

The heat sink comes with some random thermal goop.

 

Bobba Fett Original Space Gold gets rid of that with ease

 

The CPU

The AMD Phenom II 1055T six-core processor in all its glory.

 

Some quality thermal goop to choose from.. Arctic Silver 5 seem best suited for this job.

 

An extremely flashy image of the 1055T mounted.

 

The AS5 has been applied.

 

CPU all sorted.

 

Memory Bonanza

16 gigabytes of DDR3 1333MHz..

 

The memory specs.

 

All banks filled, and the board is maxed out.

 

The EVGA GTS450 in all its glory.

 

Mounted.

 

 

 

 

More images to come

 

MvH

[infected]

Endret 26. desember 2010 av [Infected]

[[Infected]]

Tid for en aldri saa liten oppdatering.. saa mye aa gjore disse dager, lite tid til noen ting

 

 

 

A poorly lit rats nest!

 

Most of the small wiring cleaned up

 

Time for some poorly lit SATA cables!

 

And even more cables..

 

General Purpose Storage

The Western Digital 500GB 7.2k RPM drive with 16MB dram cache.

These four will be configured for RAID10 for both redundancy and performance

 

The Black Edition in all its glory

 

Ready for mounting..

 

All mounted

 

The top backplane is powered from the board's SATA controller

 

Storage Controller

 

The Adaptec 1430SA PCIe 4 port SATA controller has finally arrived

 

And mounted.

 

The last few drives

Finalizing the storage for Polly.

 

Another Seagate Momentus XT 320GB hybrid drive.

 

Two A-RAM 32GB Indilinx Barefoot powered SSD's

 

SATA cables..

 

Mounting the Barefoot's in the 2.5" to 3.5" bracket.

 

Mounting the SSD's in the floppy bay.

 

The drive cable salad..

 

The LED lightshow of the Adaptec controller.

 

Polly storage system:

• 
  2x Seagate Momentus XT 320GB hybrid drives (RAID1)
  2x PQI 64GB SandForce SSD's (RAID0)
  2x A-RAM 32GB Barefoot SSD's (RAID0)
  4x Western Digital 500GB 7.2K Black Edition (RAID10)
  

 

 

The drive bays in operation.

 

The very feature-full (not to mention ultra-reliable) Gigabyte BIOS in all its glory

 

 

 

Det var alt for denne gang, naa er det paa tide aa leke litt med RAID

 

MvH

[infected]

Endret 26. desember 2010 av [Infected]

[[Infected]]

Og for de der ute som liker tall

 

Seagate Momentus XT 320GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 284 MB in 3.01 seconds = 94.47 MB/sec

raid 1: Timing buffered disk reads: 204 MB in 3.01 seconds = 67.88 MB/sec

 

PQI S528 64GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 734 MB in 3.00 seconds = 244.47 MB/sec

raid 0: Timing buffered disk reads: 1100 MB in 3.00 seconds = 366.30 MB/sec

 

WD Black 500GB (Adaptec Serial ATA II RAID 1430SA):

single: Timing buffered disk reads: 276 MB in 3.01 seconds = 91.74 MB/sec

raid 10: Timing buffered disk reads: 464 MB in 3.01 seconds = 154.17 MB/sec

 

A-RAM Pro 32GB (JMicron Technology Corp. JMB362/JMB363):

single: Timing buffered disk reads: 552 MB in 3.01 seconds = 183.57 MB/sec

raid 0: Timing buffered disk reads: 548 MB in 3.00 seconds = 182.42 MB/sec

 

(mdraid 2.6.9-x86_64, Slackware 13.1)

 

The A-RAM's raid 0 performance makes me lol

 

MvH

[infected]

Endret 6. november 2010 av [Infected]

[[Infected]]

Primaernoden er stort sett klargjort, paa tide aa sette sammen Node Array'et

 

 

Hardware

First the obligatory hardware shot!

 

 

• 
  * 8x J&W Minix 785G AM3 Micro-ITX
  * 32GB DDR2 800MHz memory (4GB per. node)
  * 2x Corsair TX650 Power Supply Units
  * D-Link 24 port Gigabit switch
  * Various models of AMD processing units
  

 

 

 

Preparations

 

All nodes have been prepped and ready.

 

Giving the nodes a full round of memtest.

 

Constructing the Stacks

The nodes will be configured in two arrays of four nodes.

 

The mounting system is constructed from stainless steel rods cut to appropriate length, coupled with pvc reticulation tubing from Bunnings Warehouse

 

Node number one.

 

Crossbars improve the structural integrity.

 

The node stack.

 

The first stack contain: 2x AMD 600e quad-cores and 2x AMD dual-core processing units.

 

Glorious 16 Gigabytes of memory

 

Constructing the secondary stack.

 

Secondary Node Array completed

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Det var alt for denne gang, flere bilder snart

 

MvH

[infected]

Endret 10. september 2011 av [Infected]

[[Infected]]

Preparing the Enclosure

 

 

 

Stripping the Antec 4U22ATX case..

 

What is the first thing to do when you receive your brand new rack mount case?

Thats right, you drill it to shits!

 

Sometimes the drill just wont 'cut it', time to bring out the heavier guns.

 

Brand new rack mount case? Not anymore!

 

The AAPEN Electrical System

Constructing the ATX power plugs by hand.

 

Just a few of all high-current connection blocks required by the project..

 

Eventually the ATX plugs are coming together.

 

Over 250 meters of assorted high-current power cable..

 

Modifying the Power Supply

Two Corsair TX650 power supply units.

 

Unboxing.

 

The TX650 unboxed.

 

Warranty void if removed.

 

If you love high end hardware, you should probably look away now.

 

Peek inside the Corsair (CWT) TX650 power supply unit.

 

Sleeved cables are just so "2009"

 

The PSU all prepped.

 

The Node Arrays..

 

The infectedtech AAPEN nodes.

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Link til Kommentartraaden

 

 

MvH

[infected]

Endret 11. september 2011 av [Infected]

[[Infected]]

Tid for en aldri saa liten update igjen, det som staar for tur naa er det sentrale strom fordelings-systemet i AAPEN, forovrig kan det nevnes at AAPEN er open-source, all programvare er lisensiert under GPL Version 3, hardware, design og hw-spesifikasjoner er lisensiert under TAPR NCL (ala. GPL for HW) og kan kopieres og benyttes fritt til alt ikke kommersielt bruk.

 

 

A package from one of my very favourite electronics vendors, Altronics.

 

 

 

Lots and lots of relays. These relays are designed specifically for use in high-current high-temperature industrial installations.

 

And yet another batch of relays.. earlier tested types and models etc.

 

Constructing the AAPEN relay clusters.

 

Wiring up the relay clusters.

 

The solder used on the relays is certainly not RoHS compliant Still, solder with a high lead content makes better joints with the copper conductor, and this is very important considering the very high currents in the system. Just got to make sure to wash hands afterwards

 

The wires used are 32x 0.2mm (17awg) and rated for use in high-current applications.

 

Heat shrink, this stuff requires a very high tempearture to shrink, using the butane iron's furnace is great for this.

 

And more wires. In total about 64 meters of high-current wiring, alongside 10-15 meters of low current wire is required for each cluster.

 

The completed relay cluster.

 

Taking the AAPEN Relay Clusters for a test run. Looks good

 

Bench testing to ensure the currents and voltages experienced in the system is within the ATX 2.2 specifications.

(Actually, all AAPEN components and wiring is rated at very minimum twice that of the ATX2.2 specs... )

 

The voltages look really good as well

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Det var alt for naa, vil ogsaa minne om kommentartraaden.

 

MvH

[infected]

Endret 10. september 2011 av [Infected]

[[Infected]]

Blir bare en liten update denne gangen, paa foresporsler blir denne og neste

update av det mer tekniske slaget, der jeg vil forklare litt mer bagrunnen i hvordan

AAPEN virker og hva jeg har tenkt ved utviklingen av plattformen.

 

Denne updaten er dedikert til programvaren som kjores paa AAPEN, programvaren er skrevet

i C og Perl, der C er benyttet til de underliggende funksjonslaget, mens Perl er brukt til

selve interfacet i AAPEN. AAPEN vil bli sluppet i en Linux distro (kalt AAPEN Linux) basert

paa Slackware Linux, selv om programvaren i seg selv kan kjores paa de aller fleste POSIX

plattformer (*nix) med stotte for de nodvendige Perl modulene, samnt stotte for ANSI C bibliotekene.

 

 

 

 

 

 

 

 

 

 

 

 

 

MvH

[infected]

[[Infected]]

Siden det har vaert saa stille her en stund.. saa noen teasers

 

upgrades

 

the AAPEN Linux Development Platform

 

Forhaapentligvis, kanskje noen updates om ikke saa lenge!

 

MvH

[infected]

Endret 24. juli 2011 av [Infected]

[[Infected]]

Lang tid med lite updates her, men naa som semesteret endelig har startet opp igjen etter ferien

er det kanskje paa tide med en aldri saa liten update fra denne kanten ogsaa

 

 

 

 

The stacks..

 

Preparing the enclosure

 

Test fit

 

Top view

 

Front view

 

Lots of wires

 

Attaching the quick connectors

 

Cheers!

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

Endret 10. september 2011 av [Infected]

[[Infected]]

Tid for enda en aldri saa liten oppdatering..

 

 

 

The 4RU enclosure, prepped and ready.

 

Mounting the Relay control board, the board used is from Denkovi Electronics.

 

Wiring up.

 

The high current industrial realy cluster. there will be a considerable amount of Amperes passing thru here..

 

High current wiring. everything is rated for the load, and then some.

 

Sampling the layout.

 

So many wires.. where will they all go? I forget.

 

Top view of the relay cluster.

 

Wiring up the control board.

 

Selfless self promotion.

 

All work and no beer, well is no way to work at all is it. Cheers!

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

Kommentartraaden

Endret 24. august 2011 av [Infected]

[[Infected]]

En aldri saa liten smakebit paa ting som kommer..

 

 

 

 

 

 

 

 

 

MvH

[infected]

[[Infected]]

En aldri saa liten oppdatering, denne gangen er det balanserings-systemet som staar for tur, dette sorger for aa holde spenningene i systemet stabile, selv under en variabel last.

 

 

 

The AAPEN Power Balancing system, also known as the AAPB. Special thanks to Khaled Daabaj, phd computer engineering, for helping me with the math on this one.

 

The capacitors..

 

Wiring up

 

Heatshrinking

 

The completed product.

 

Mounted

 

All hardware and electrical design of this project is licensed under the TAPR Open Hardware NCL. (Which mean you may copy freely; but not use for any commercial use without explicit permission to do so, any non-commercial use follows standard Open Source conventions. Free as in speech, not as in beer.)

 

 

 

 

MvH

[infected]

[[Infected]]

En aldri saa liten update, denne gangen med fokus paa testing

 

 

 

 

Inspection Guidelines..

 

Datablad er alltid morsomt

 

Ser fint ut

 

Fint, fint

 

Varmt

 

Fint

 

IB kontrolleren markerer som den varmeste delen av noden

 

Stromfordelingssystemet varmeste delen av arrayet

 

Point of reference

 

 

 

 

og helt til slutt, en liten video tatt under testen

http://www.youtube.com/watch?v=27rT1P35sTw

 

Mvh

[infected]

Endret 26. oktober 2011 av [Infected]

[[Infected]]

En moderne (single-rail) ATX stromforsyning

 

Diagram over stromsystemet

 

Diagram over logikken

 

 

Mvh

[infected]

Endret 7. desember 2011 av [Infected]

[[Infected]]

En liten update denne gangen, for det meste random bilder tatt under forbedredelse for ny kabling samt installasjon av IB etc.

 

 

 

 

Campbeltown Loch, only good for killing braincells and degreasing CPU's.

 

Diverse maalinger for kanalene kan tas opp for enda mer testing :!:

 

Stromsystemet og strombalanseringssystemet

 

Bilde foran av nodene for modifiseringer av stack'ene

 

En Node Stack (stack #2) slik de ser ut for modifiseringer.. (1.0a)

 

Nevermind har ogsaa faatt en aldri saa liten oppgradering. Fra NAS til NAS + SAN

Adaptec SAS 31605 saa naa blir visst filserveren med i AAPEN ogsaa

 

Oppgradering til infiniband! tonnevis med bilder kommer

 

Er slutten av enda ett semester naa (AAPEN has reached ALPHA!), etter siste innspurt med dokumenter og last-minute edits seff, saa naa god ferie med tid til aa jobbe litt mer med koden, samt kanskje poste noen flere updates ogsaa :!:

 

Og helt til slutt, tro det eller ei, AAPEN har naadd ALPHA!!

 

 

 

 

Har ogsaa noen heller spennende ting linet upp paa hardware fronten for semesteret som kommer, saa om du ikke har noe bedre ting aa ta deg til, folg med videre!

 

Forslag eller tips? Kommentartraaden

 

Mvh

[infected]

Endret 6. januar 2012 av [Infected]

[[Infected]]

Oppgradere nodene til infiniband, part 1;

 

Forste som trengs er ett par PCIe ribbon kabler

 

En liten haug med IB HCA'er

 

Preparere HCA'ene

 

Prepare nodene

 

Litt prototyping

 

Litt forstyrrelser

 

Og demontering..

 

Stay tuned!

 

Mvh

[infected]

Endret 6. januar 2012 av [Infected]

[[Infected]]

Oppgradere nodene til infiniband, part 2;

 

 

Begynne aa ta ting fra hverandre

 

IB kontrollerene klar til montering

 

Tilpasse IB kablene, dette kommer til aa bli trangt

 

Annen vinkel

 

Testing, testing og mere testing. En skikkelig testplan, er coupe de grace, ivertfall i folge min supervisor

 

Forste node i stack'en preparert, klargjore neste

 

Ett detaljskudd

 

Ikke helt enkelt aa faa bilde av IB kontrolleren gjemt bort der

 

Bilde fra toppen, detaljarbeid dette

 

Bilde fra front

 

Test, test og mere test

 

Saa de obligatoriske timene med bench og burn testing, alt maa selvfolgelig logges

 

Mvh

[infected]