Jump to content

2x 96TB ZFS Build log + 10g goodness

cookiesowns
By cookiesowns
in Build Logs
 Share
Posted

Hey Guys!

 

Table of contents:

 

July 6th. 2014 - Original post with pictures

 

July 9th. 2014 - Picture updates of pc components, APC Rack & more!

 

Anywho:

 

New to LTT forums, but I've been watching Linus's videos for quite some time now. I've been working in IT for awhile, and just recently I get to implement an IT upgrade for work.

 

Our company does quite a bit of digital media, such as photography, videos, and educational content. Currently we operate on a few synologies as well as some off-site backup routines, but we are growing far quicker than the synologies can handle, and we needed a 10G upgrade anyways.

 

So here we go:

 

ZFS #1

CPU: Intel Xeon E5-1620V2

RAM: 4x16GB ECC 1866 REG

HDD: 24x 4TB Hitachi NAS drives in 4x 6 drive Z2

MB: Supermicro X9SRH-7TF

Chassis: Supermicro SC846BA-R920B

HBA: LSI 9207-8i x2 + onboard LSI 2308

SSD: 2x Intel S3500 120GB for boot + l2arc/ZIL testing

OS: Maybe FreeNAS, undecided yet

 

ZFS #2 ( offsite backup server )

CPU: Intel Xeon E5-1620V2

RAM: 4x16GB ECC 1866 REG

HDD: 24x 4TB Hitachi NAS drives in 2x 11 drive Z3

MB: Supermicro X9SRH-7TF

Chassis: Supermicro SC846BA-R920B

HBA: LSI 9207-8i x2 + onboard LSI 2308

SSD: 2x Intel S3500 120GB for VM's

OS: Promox w/ ZoL or FreeNAS

 

Networking:

master workstations will have X540-T2 NICS

Pfsense on a 5018A-MHN4 box which has an Intel Avoton C2758

Juniper EX3300-48T as "core"

Netgear XS712T as 10G distribution with 2x 10G SFP+ uplinks to the juniper switch

 

Rack:

2 cyberpower smart app 2150VA UPS units

APC AR3100 42U standard rack

 

Lightroom Build:

Doing another editing machine build which will most likely end up being the benchmark rig, if not, we will use the 4930K rig below

Intel 4790K

G.SKILL 1866 C8 2x8GB

Intel 730 240GB

Asus Maximus VII Hero

Nvidia GTX 660

AX760

Corsair 450D

 

Benchmark rig:

Intel 4930K @ 4.4

Asus X79-Deluxe

32GB G.Skill 1866 C8 4x8GB

1x Intel X540-T2

1x LSI 9207-4i4e <- This is for initial testing of drives + testing out an HP 6250 LTO-6 tape drive.

 

 

Pictures to come as parts come in as I've created this thread to allow easier posting from work!

 

Any questions? Picture/benchmark requests? Throw them in the thread!

 

 

-Update 7.3.2014 -

More stuff came in! Phone pics in the meantime:

 

Intel X540-T2 Copper NIC

 

2IT4Mgtl.jpg

 

 

OM3 LC/LC Fiber patch cord. Got them from mono price, and they seem to use corning fiber which is great

Fwevki5l.jpg

 

4x SFP+ 10G optics from fiberstore

 

2 Netgear coded optics, and 2 juniper coded optics. Curious to see how they fare. They were $25 each for 10G-SR optics. Juniper optics would be around $700-$1500 depending on where you get it. HUGE difference! If you have a jcare contract, then yes you would want to go with Juniper original optics, otherwise, just buy some spares.

 

iifSfAul.jpg

 

52 Hitachi Deskstar NAS drives were ordered, here you see about 32 drives.

 

ENMrRbfl.jpg

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

mind blown

 

i'm assuming about gaming possibility in a server build! 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

holy shit subbed :D 

If you tell a big enough lie and tell it frequently enough it will be believed.

-Adolf Hitler 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Subbed 

 

want to build a storage rig someday tho my current rig coud be classified as storage (19tb not including os drive)

Forum Links - COC I FAQ I

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Subbed.

 

Don't you need 1GB of ram per 1TB of HDD?

 

It's a general rule of thumb, but does not need to be followed to the dotted i. 64GB of RAM should be enough for what I'm using it for.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

OMG I WANT MORE!

There are 10 types of people in the world: Those who understand binary, and those who don't.

Just some helpful stuff: You're - You are, Your - Your car, They're - They are, Their - Their car, There - Over there.

 

Folding @ Home Install Guide and Links | My Build

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

All of my want...

*froth*

                    Fractal Design Arc Midi R2 | Intel Core i7 4790k | Gigabyte GA-Z97X-Gaming GT                              Notebook: Dell XPS 13

                 16GB Kingston HyperX Fury | 2x Asus GeForce GTX 680 OC SLI | Corsair H60 2013

           Seasonic Platinum 1050W | 2x Samsung 840 EVO 250GB RAID 0 | WD 1TB & 2TB Green                                 dat 1080p-ness

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Subbed. Let us know how hot the heatsinks on the X540-T2s get, I'm debating getting one but am unsure of the airflow requirements.

 

Don't you need 1GB of ram per 1TB of HDD?

It is generally recommended, but not required. The only downside will be reduced read/write caching in RAM, but large media files wouldn't get cached in RAM anyways, because large sequential transfers get written directly to disk.

I do not feel obliged to believe that the same God who has endowed us with sense, reason and intellect has intended us to forgo their use, and by some other means to give us knowledge which we can attain by them. - Galileo Galilei
Build Logs: Tophat (in progress), DNAF | Useful Links: How To: Choosing Your Storage Devices and Configuration, Case Study: RAID Tolerance to Failure, Reducing Single Points of Failure in Redundant Storage , Why Choose an SSD?, ZFS From A to Z (Eric1024), Advanced RAID: Survival Rates, Flashing LSI RAID Cards (alpenwasser), SAN and Storage Networking

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Subbed. Let us know how hot the heatsinks on the X540-T2s get, I'm debating getting one but am unsure of the airflow requirements.

 

It is generally recommended, but not required. The only downside will be reduced read/write caching in RAM, but large media files wouldn't get cached in RAM anyways, because large sequential transfers get written directly to disk.

good to know about the ram thing

Forum Links - COC I FAQ I

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Small updates today:

 

APC Netshelter SX42U rack came in. Apparently dell now rebrands / sells these.

AMPqqd3l.jpg

 

la3fV5Xl.jpg

 

 

PC Goodies came in too! Still missing H110 & the processor

 

EKJi7yll.jpg

 

lJ2H57sl.jpg

 

 

ihgCkB6l.jpg

 

oLaPWqWl.jpg?1

 

 

 

 

Subbed. Let us know how hot the heatsinks on the X540-T2s get, I'm debating getting one but am unsure of the airflow requirements.

 

 

Will do. From what I can tell, I don't think they will get "too" hot, at least not anywhere near LSI2208 based raid cards get. I have a feeling they will run toastier than your regular card though, as the super micro branded X540-T2's have an active fan on it.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

omg. please. more!

There are 10 types of people in the world: Those who understand binary, and those who don't.

Just some helpful stuff: You're - You are, Your - Your car, They're - They are, Their - Their car, There - Over there.

 

Folding @ Home Install Guide and Links | My Build

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Sorry for the shortage of updates guys.

 

CPU & H110 came yesterday, so now the PC is built and ready. Mac OC was 4.7Ghz, Max "Stable" Oc was 4.6Ghz. Will be running 4.5Ghz daily @ 1.25V.

 

oYEy4KFl.jpg

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

What did I just wake up to?

 

tumblr_mov4f86UZI1rpnk51o1_500.gif

love that episode :D:D:D:D

Gaming Computer | Case: CM Storm Scout II | MB: Asus P8Z77-V LK | CPU: i5 2320@3,6ghz | GFX:  MSI GTX 770 Gaming | RAM: 8gb DDR3 @ 1333mhz | Storage: 1TB Seagate Barracuda | Monitor: 1x Asus VG248QE @ 144Hz , 1x Samsung SyncMaster SA300 @ 77hz OC'd 


Other Device's | Phone: Samsung Galaxy S3 LTE@OmniROM | Tablet: Samsung Galaxy Tab 2 10.1@OmniROM | Laptop: Lenovo ThinkPad T61 Win 8.1

Link to comment
Share on other sites
Link to post
Share on other sites
  • 2 weeks later...
Posted
  • Author

Some photos.

 

Didn't grab too many photos of the first server as had a deadline to meet. But I should be taking more detailed shots of the second servers build process. 

 

Tray processors straight from Intel's Costa Rica

 

2vAuneUl.jpg

 

 

4x Intel S3500 120GB SSD's

 

yqznqpml.jpg

 

 

Typical supermicro packaging. X9SRH-7TF

 

2MT6DjOl.jpg

 

This is new... Supermicro now comes with a color quick start guide. Very helpful!

 

ZoYkoawl.jpg

 

 

Useless generic red sata cables. Anyone want these? I still have like 50 of them left over from previous super micro installations

 

LaGV82hl.jpg

 

The actual board! Latest BIOS & IPMI firmware but the LSI firmware was on 13 IR, which I think is pretty old. Flashed to R16 IT in prep for FreeNAS. Supermicro did also release a R19 build, so I might try flashing that too.

 

dltCvCql.jpg

 

Hynix 16GB 1866 ECC REG dims that are on the Supermicro QVL

 

GFvBG6tl.jpg

 

 

 

 

The LSI HBA's!

fcRbtQvl.jpg

 

 

Supermicro Avoten Superserver barebones. I only needed to install ram and that was it! Also nice to see how super micro does the wiring too.

 

 

R7aLmRBl.jpg

 

 

Here's how the rack is populated in the meantime, and one of the 4U servers built. I've ran a windows server 2012 installation, and created a pool with all 24 drives. Speeds were interesting. 1.2GB/s read/write sequential, and random performance was basically 1-2 drives at most. Will try some more server 2012 testing before breaking it down to run bad blocks for 1-2 passes.

 

Memtest on both servers passed with flying colors, literally.. since the EFI memtest is colorful

 

 

AaI6iFOl.jpg

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I'm done.

Like what the f*ck? That's alot of space

 

Spoiler

i5 4670k, GTX 970, 12GB 1600, 120GB SSD, 240GB SDD, 1TB HDD, CM Storm Quickfire TK, G502, VG248QE, ATH M40x, Fractal R4

Spoiler

i5 4278U, Intel Iris Graphics, 8GB 1600, 128GB SSD, 2560x1600 IPS display, Mid-2014 Model

Spoiler

All the parts are here, just need to get customized cords to connect the motherboard to the front panel.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

This is awesome.

 

If you get a chance, please run some storage performance benchmarks once you have the ZFS fully setup :)

Will you be running NFS? If so, could you share any special configurations you end up using to optimize for 10Gb?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

This is awesome.

 

If you get a chance, please run some storage performance benchmarks once you have the ZFS fully setup :)

Will you be running NFS? If so, could you share any special configurations you end up using to optimize for 10Gb?

 

 

Unfortunately I won't be running NFS. It'll be mainly SMB/CIFS and one iSCSI share on a mirrored vddv. 

Sure, I'll let you know my findings on 10Gb. I've decided to not bother with server 2012 now as well, so sorry guys, no results for ya!

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Wait a second... Are you using a 20 pin in the 24pin slot... Does that work?

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Build Logs

×