Jump to content

APOLLO (2 CPU LGA1366 Server | InWin PP689 | 24 Disks Capacity) - by alpenwasser [COMPL. 2014-MAY-10]

alpenwasser
 Share
Posted
  • Author

That's a lot of cables.

 

Yup, it is indeed. Can't be helped really until we get to wireless SATA. :D

 

A hot glue gun can help, especially to keep cables out of the fans.

Hm, haven't considered that yet. The fans aren't too much of a problem,

the cables are pretty stiff and don't really have much play to flap

around.

Ideally I'd buy some longer cables to have some spare cable length

for optimal routing, but since these cables aren't exactly cheap I'm

not quite sure if I'll go through with that (after all, there won't

be a window, and they aren't all that cheap here).

Thanks for the tip though, I'll keep it in mind. :)

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

 

Yup, it is indeed. Can't be helped really until we get to wireless SATA. :D

 

Hm, haven't considered that yet. The fans aren't too much of a problem,

the cables are pretty stiff and don't really have much play to flap

around.

Ideally I'd buy some longer cables to have some spare cable length

for optimal routing, but since these cables aren't exactly cheap I'm

not quite sure if I'll go through with that (after all, there won't

be a window, and they aren't all that cheap here).

Thanks for the tip though, I'll keep it in mind. :)

Hmm, that sucks. I personally an averse to making anything permanent (hot glue is permanent mostly, or it's temporary but leaves a mess). 

I use www.monoprice.com for all my cable stuff. I find they have the absolute best quality-price ratio, but that doesn't help you being in Switzerland and all. :( 

† Christian Member †

For my pertinent links to guides, reviews, and anything similar, go here, and look under the spoiler labeled such. A brief history of Unix and it's relation to OS X by Builder.

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

 Yup, it is indeed. Can't be helped really until we get to wireless SATA. :D

Wireless SATA would be awesome!

Rig CPU Intel i5 3570K at 4.2 GHz - MB MSI Z77A-GD55 - RAM Kingston 8GB 1600 mhz - GPU XFX 7870 Double D - Keyboard Logitech G710+

Case Corsair 600T - Storage Intel 330 120GB, WD Blue 1TB - CPU Cooler Noctua NH-D14 - Displays Dell U2312HM, Asus VS228, Acer AL1715

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Hmm, that sucks. I personally an averse to making anything permanent (hot glue is permanent mostly, or it's temporary but leaves a mess). 

I use www.monoprice.com for all my cable stuff. I find they have the absolute best quality-price ratio, but that doesn't help you being in Switzerland and all. :(

I did found four of the SAS->SATA breakout cables on eBay for a very

good price, maybe I'll get lucky and find some more offers that ship

to me within a reasonable timeframe. But in all honesty: Having a

slight cable mess in the M/B compartment is not such a huge deal in

this case as said. It does bug me a tiny bit, be meh, it's not so

important in this build. Maybe I'll buy more cables, maybe not, it

really depends on price and whether or not I'm motivated enough to

redo the cabling in the disk compartment (which is mostly done).

We shall see... ;)

 

Wireless SATA would be awesome!

IKR!? :D

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Storage Topology & Cabling


Storage Topology


In case you can't read the text, the full res version should
be more easily readable.

(click image for full res)
aw--apollo--2014-04-13--storage-topology

The   idea  behind   the  storage   topology  is   based  on
@wpirobotbuilder's  post  about  reducing single  points  of
failure. Any one of the three LSI controllers can fail and I
still have all my data available.


You'll  see  below  that  I haven't  yet  gotten  around  to
installing the Velociraptor.

I use coloured zip ties to mark the cables that go to the
different controllers.

BLUE   = controller 0
YELLOW = controller 1
GREEN  = controller 2


Tidiness

There isn't really any space to hide the cables, so this was
rather  tricky  and  required  three attempts  until  I  was
satisfied with the result. In the  end I hid the extra cable
behind the triple  fan unit, good thing they're  38 mm fans,
which makes the space behind them just about large enough to
fit the extra cable bits.

The power cables for the disks are two cables that came with
the PSU  and onto  which I  just put  a lot  more connectors
while  taking off  the stock  connectors because  those were
neither placed  in the correct  locations nor facing  in the
right direction.


Looks harmless, right? Yeah...

(click image for full res)
aw--apollo--2014-04-13--01--cables-harml


And the disks:
(click image for full res)
aw--apollo--2014-04-13--02--disks.jpeg


OK then, first try:
(click image for full res)
aw--apollo--2014-04-13--03--all-cables.j

I soon realized that this  wasn't going to work. The problem
was that  I had the  disks arranged in  the same way  as the
will be  set up  in the  storage pool  layout, so  the disks
which go into the same  storage pool were also mounted below
each  other.  Sounds  nice in  theory,  but if  you want  to
have  disk from  each pool  distributed among  the different
controllers, you'll get quite the cable mess.

(click image for full res)
aw--apollo--2014-04-13--04--cabling-roun

(click image for full res)
aw--apollo--2014-04-13--05--cabling-roun


Second Try

Next try, this time I arranged  the disks to that the cables
to the controllers could be  better laid out. Since I wanted
to set up  all the cables for all the  disk slots, even ones
that will  stay empty for  now, I  had to shuffle  the disks
around when laying out the cables.

(click image for full res)
aw--apollo--2014-04-13--06--cabling-roun

(click image for full res)
aw--apollo--2014-04-13--08--cable-hydra.

(click image for full res)
aw--apollo--2014-04-13--09--cable-crossr


Better. But I still wasn't quite happy, mainly because...

(click image for full res)
aw--apollo--2014-04-13--10--cables-done.

(click image for full res)
aw--apollo--2014-04-13--11--disks-distri

(click image for full res)
aw--apollo--2014-04-13--12--cables-detai


... of this:

(click image for full res)
aw--apollo--2014-04-13--13--cables-nest.


Third Try


This time  I made sure the  cables stayed tidy on  both ends
while hiding  the mess  (which cannot  be avoided  since all
cables are the same length but lead to different end points,
obviously) behind the triple fan unit.

(click image for full res)
aw--apollo--2014-04-13--15--hiding-space


The loop of extra cable length for the top cable loom:

(click image for full res)
aw--apollo--2014-04-13--16--hiding-space


And the cable loom for controller 0, from the disk side...

(click image for full res)
aw--apollo--2014-04-13--17--cable-loom-c


and the M/B side. Much better IMHO. :)

(click image for full res)
aw--apollo--2014-04-13--18--cable-loom-c


The bottom controller had a bit more extra cable length to hide, so
that part is a bit messier.

(click image for full res)
aw--apollo--2014-04-13--19--cable-loom-c

And the middle one:
(click image for full res)
aw--apollo--2014-04-13--20--cable-loom-c


Tada! While not perfect (I'd need  longer cables for that to
make cleaner runs,  but I'm not buying more  cables just for
the sake of that for a  build that has a closed side panel),
with this iteration of my cabling I'm now rather happy:

(click image for full res)
aw--apollo--2014-04-13--21--disk-rack-ca

(click image for full res)
aw--apollo--2014-04-13--22--disk-rack-ca

(click image for full res)
aw--apollo--2014-04-13--23--disk-rack-ca


And the other side. Much better than before methinks. :)

(click image for full res)
aw--apollo--2014-04-13--24--mb-cmpt-over

(click image for full res)
aw--apollo--2014-04-13--25--mb-cmpt-clos


The SATA cable for the system SSD:

(click image for full res)
aw--apollo--2014-04-13--26--system-drive


And the controller LEDs when there's some activity:

(click image for full res)
aw--apollo--2014-04-13--27--controller-l


Now  if you'll  excuse me,  there's a  dinner waiting  to be
cooked. :)


Cheers,
-aw
 

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I must admit, as I was scrolling down through those pictures, my expression went from:

 

Oh no what the heck is he doing??  This is nowhere near as neat as I was expecting.

 

To:

 

Mother of god... It's beautiful!

 

Top notch work on those SATA cables man, looks like something straight out of a high end data centre.

Finished Projects: Loramentum, Vesper, Moderne, Exsectus, Aetos

Current Project: Parvum Argentum

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Veeery nice :)

"Unofficially Official" Leading Scientific Research and Development Officer of the Official Star Citizen LTT Conglomerate | Reaper Squad, Idris Captain | 1x Aurora LN


Game developer, AI researcher, Developing the UOLTT mobile apps


G SIX [My Mac Pro G5 CaseMod Thread]

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Looks cool and great work!

 

Now, don't mean to be a downer after all your hard work but what happens when you need to remove a drive :( Not that I want you experience that but...

 

I have been reading/looking at your setup and thinking about your layout and restrictions, I think you've done a masterful job and design and build. I can only envision some sort of back plane that might improve it but would limit cooling as most back planes are not very open. With the disk layout you either have to remove the drives or the fans to remove/add a drive.

 

All in all that is the best for the layout.

 

Will you run any benchmarks on the zpools? Would be cool to see your benchmarks for this setup.

I roll with sigs off so I have no idea what you're advertising.

 

This is NOT the signature you are looking for.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

I must admit, as I was scrolling down through those pictures, my expression went from:

 

 

To:

 

 

 

Top notch work on those SATA cables man, looks like something straight out of a high end data centre.

 

Haha, yeah I had my doubts whether or not I was ever going to get the

cables tidy to be honest. There's a reason it took me three tries after

all.

 

It could still be improved probably, but I see no way to do that without

buying longer cables, which isn't really something I'm willing to do.

Thanks, in any case! :)

Veeery nice :)

Thank you! :)

 

 

Looks cool and great work!

 

Now, don't mean to be a downer after all your hard work but what happens when you need to remove a drive :( Not that I want you experience that but...

 

I have been reading/looking at your setup and thinking about your layout and restrictions, I think you've done a masterful job and design and build. I can only envision some sort of back plane that might improve it but would limit cooling as most back planes are not very open. With the disk layout you either have to remove the drives or the fans to remove/add a drive.

 

First of all: thanks for the kind words! :)

I actually considered getting a backplane, either as a spare part for

a proper server, or a server pull part, but the ones that would ship

to me were neither the right dimension (too large), nor within an

acceptable price range unfortunately.

However, swapping out a drive has been taken into consideration, and

while it's not exactly hot swap, downtime shouldn't be more than a few

minutes (which is acceptable for a home and small business file server

in our case). The steps required are as follows:

  • Power down and open the server on the back side for accessing

    the disks (the side where the cables and connectors are located).

  • Disconnect as many data and power connectors as needed to be

    able to get out the drive which needs to be replaced (this sounds

    like a rather big deal with so many drives, but it's actually a

    rather quick and painless procedure because the power cables are

    basically just two long pieces, and the data cables are nicely

    zip tied together, meaning there won't be a big mess when you

    disconnect it all).

  • Take new drive and four screws (I have set aside enough screws

    already and will have them handy so that I don't have to search

    for them in that moment, and since the server is located in our

    workshop, a screwdriver should be easy enough to find ;)) as

    well as some Loctite, put Loctite on screws, mount screws to HDD

    with enough thread to spare so that the disk can still slide in.

    The Loctite is needed because the screws rotate when you slide

    the disk in and out.

  • Slide disk in, make sure it's locked into place.
  • Reconnect everything (again, pretty simple since every connector

    will be more or less in the right place already due to the premade

    cable looms).

  • Boot up. :)
As said, not exactly hot-swappable, but as long as I don't have to

swap out drives on a daily basis it's actually a pretty comfortable

system to use. :)

All in all that is the best for the layout.

 

Will you run any benchmarks on the zpools? Would be cool to see your benchmarks for this setup.

I haven't really run proper benchmarks, and I'm still working out some kinks

(mainly related to the virtual machine device drivers). At the moment I'm not

really getting that impressive performance, ~60 to 90 MB/s writing, depending

on the situation. However, considering I'm running non-striped RAIDZ2 pools,

that's actually not too bad I think. As far as I understand, you could get the

performance of a single drive, maximum, and the parity calculations for double

parity cost quite a bit of performance according to the ZFS docs I've read, so

I'm quite happy for now (although of course still looking for ways to improve).

I've thought about caching devices for my personal data pool, but I doubt I'll

be implementing that since performance is actually perfectly sufficient at the

moment already.

When the moment comes to buy some more drives (which might not be off too far

into the future actually), I will play around a bit with those, see how the

setup performs, but all the drives I currently have I actually need for production,

so I can't really do much playing around.

So: All in good time. :)

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Wow, this build is absolutely beautiful! I love nice server builds in general, but this just blew me away. Great job on this man!

"Her tsundere ratio is 8:2. So don't think you could see her dere side so easily."


Planing to make you debut here on the forums? Read Me First!


unofficial LTT Anime Club Heaven Society

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Holy Jesus! I'm in awe that you need this monster of a file server just for home use (and other server applications too I'd assume :lol: )

Excellent work man, Might just take crown as the best home server on the forums (if your ZEUS didn't have it already ;) )

                    Fractal Design Arc Midi R2 | Intel Core i7 4790k | Gigabyte GA-Z97X-Gaming GT                              Notebook: Dell XPS 13

                 16GB Kingston HyperX Fury | 2x Asus GeForce GTX 680 OC SLI | Corsair H60 2013

           Seasonic Platinum 1050W | 2x Samsung 840 EVO 250GB RAID 0 | WD 1TB & 2TB Green                                 dat 1080p-ness

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Wow, this build is absolutely beautiful! I love nice server builds in general, but this just blew me away. Great job on this man!

 

Thanks, much appreciated! :)

 

My, what an excellent suggestion! :D

 

 

Holy Jesus! I'm in awe that you need this monster of a file server just for home use (and other server applications too I'd assume :lol: )

Excellent work man, Might just take crown as the best home server on the forums (if your ZEUS didn't have it already ;) )

Haha, thanks for the compliments. Yeah, it's basically the file server for my

peronal data, our media library and my dad's business data.

And yes, while it's not the prettiest thing out there, I do quite like looking

at it personally. :)

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

<---- shows how i feel cuz i cant have this too :)

"Unofficially Official" Leading Scientific Research and Development Officer of the Official Star Citizen LTT Conglomerate | Reaper Squad, Idris Captain | 1x Aurora LN


Game developer, AI researcher, Developing the UOLTT mobile apps


G SIX [My Mac Pro G5 CaseMod Thread]

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Any one of the three LSI controllers can fail and I

still have all my data available.

Yay redundancy of all kinds!

 

That is a seriously full system.

I do not feel obliged to believe that the same God who has endowed us with sense, reason and intellect has intended us to forgo their use, and by some other means to give us knowledge which we can attain by them. - Galileo Galilei
Build Logs: Tophat (in progress), DNAF | Useful Links: How To: Choosing Your Storage Devices and Configuration, Case Study: RAID Tolerance to Failure, Reducing Single Points of Failure in Redundant Storage , Why Choose an SSD?, ZFS From A to Z (Eric1024), Advanced RAID: Survival Rates, Flashing LSI RAID Cards (alpenwasser), SAN and Storage Networking

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

A truly epic system, how much storage are we talking here? I may have overread it somewhere, but we're probably looking in the neighborhood of like 30TB?! 

That's just insane :D

phanteks enthoo pro | intel i5 4690k | noctua nh-d14 | msi z97 gaming 5 | 16gb crucial ballistix tactical | msi gtx970 4G OC  | adata sp900

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Yay redundancy of all kinds!

 

That is a seriously full system.

Haha, yeah I'm quite happy with it, and it still has more than 10

disk slots free. :D

 

 

A truly epic system, how much storage are we talking here? I may have overread it somewhere, but we're probably looking in the neighborhood of like 30TB?! 

That's just insane :D

At the moment it's 29 TB raw storage, so yeah, you're not too far off. ;)

Usable capacity when deducting parity is 17 TB (12 TB on the media pool,

4 TB on my personal data pool, and 1 TB on my dad's business pool).

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

29000 GB, that's just stupidly insane :D

phanteks enthoo pro | intel i5 4690k | noctua nh-d14 | msi z97 gaming 5 | 16gb crucial ballistix tactical | msi gtx970 4G OC  | adata sp900

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

29000 GB, that's just stupidly insane :D

Well, my personal goal is to have this baby breach 100 TB at some point,

but I reckon that'll have to wait for a few years. :D

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Well, my personal goal is to have this baby breach 100 TB at some point,

but I reckon that'll have to wait for a few years. :D

 

How many 3.5" spots do you have? If the 6TB drives become affordable anytime soon, then I don't think this'll be much of a problem :D

phanteks enthoo pro | intel i5 4690k | noctua nh-d14 | msi z97 gaming 5 | 16gb crucial ballistix tactical | msi gtx970 4G OC  | adata sp900

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

How many 3.5" spots do you have? If the 6TB drives become affordable anytime soon, then I don't think this'll be much of a problem :D

24 slots, so yeah, no problem. But I doubt they'll become affordable any time

soon, I reckon it will be a year, probably two.

Honestly though, I'd be much more interested in 5 TB drives. 1 TB platters have

been on the market for quite a while, and the technology to make a 5-platter drive

is also tried and tested (some 4 TB drives use 5 platters with 800 GB each, others

use 4 platters with 1 TB each from what I've read), so in all honesty, just give

me a drive with 5 platters of 1 TB each for a decent price and I'll be happy,

the technology to do that is already here, just freaking make that drive FFS!

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Haha, yeah I'm quite happy with it, and it still has more than 10

disk slots free. :D

 

 

Before you get real data are you going to test the failure modes?

I roll with sigs off so I have no idea what you're advertising.

 

This is NOT the signature you are looking for.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Before you get real data are you going to test the failure modes?

Yes, when I buy more disks I will definitely do that. Failure tests,

benchmarks, whatever comes to mind. Might still be a while until then

though because monies are currently being a bit elusive, those buggers. ;)

But I'm very much looking forward to that actually.

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

that's allot of porn.

Nah, storing porn locally isn't really cost-effective with all the free

porn you can get on the internet these days. Most of the storage will

go to extremely high resolution images of empty sheets of paper.

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Build Logs

×