Looking for a high-capacity storage server build.

[Crcoli737]

Budget (including currency): $8000-10000 US (Flexible)

Country: United States

Games, programs or workloads that it will be used for: 

Other details (existing parts lists, whether any peripherals are needed, what you're upgrading from, when you're going to buy, what resolution and refresh rate you want to play at, etc): 

 

Looking for a possible server for a local photography business. Storage is a must! Looking for some sort of NAS drives, probably in the 500-600 TB range. Don't need super fast read times. However, I plan on running TrueNAS on it. Right now, we are upgrading from about 300 TB of external USB hard drives. Plan on accessing the server from both Windows and Mac devices. Plan on buying withing the next 6-8 months or so. I would also like to be able to edit using Photoshop directly off of the server, not very common but from time to time. I'm not going to provide a parts list as I would like to see what you all come up with. It is also important that it stays online 24/7. Average image size is about 20-30 MB. 

 

Let me know if you have any questions!

[Electronics Wizardy]

This kinda feels like a hire it out type of thing.

 

But I'd look at the supermicro or 45 drives dense storage setups. They both should have top loader 4u system that can hold lots of drives and well over 600TB of data. 

 

Most of these systems are very noisy, power hungry(probably need to plan the hvac), and often need 200-240v to operate. The rack also needs to support the weight and depth of these systems. 

 

Do you have a server rack to put this in?

 

Make sure to have a backup plan for this box.

[Crcoli737]

9 minutes ago, Electronics Wizardy said:

But I'd look at the supermicro or 45 drives dense storage setups. They both should have top loader 4u system that can hold lots of drives and well over 600TB of data. 

 

All of the 45 drives systems we looked at seem to be on the expensive side of our budget. Looking to see if there are any cheaper option. 
 

10 minutes ago, Electronics Wizardy said:

Most of these systems are very noisy, power hungry(probably need to plan the hvac), and often need 200-240v to operate. The rack also needs to support the weight and depth of these systems. 

We are in a professional building, which has 120v wiring. Would like to stick with that if possible. The HVAC we have is great. Especially in our mechanical/network room. 
 

11 minutes ago, Electronics Wizardy said:

Do you have a server rack to put this in?

Yes. We have a 42U rack to put it in. 
 

12 minutes ago, Electronics Wizardy said:

Make sure to have a backup plan for this box.

We are looking at some options right now. Hence why we are looking at a larger window for obtaining our server (6-8 months)

[Electronics Wizardy]

8 minutes ago, Crcoli737 said:

We are looking at some options right now. Hence why we are looking at a larger window for obtaining our server (6-8 months)

Tapes make a good amount of sense at that scale. Or cloud backups.

 

9 minutes ago, Crcoli737 said:

All of the 45 drives systems we looked at seem to be on the expensive side of our budget. Looking to see if there are any cheaper option. 

Would this system be cheaper? https://store.supermicro.com/us_en/4u-superstorage-ssg-6049p-e1cr45h.html

 

I'd get some quotes from supermicro for a system like this

 

What drive config are you planning?

[Crcoli737]

14 minutes ago, Electronics Wizardy said:

I'd get some quotes from supermicro for a system like this

I’ll definitely look into it and shoot them an email. 
 

14 minutes ago, Electronics Wizardy said:

What drive config are you planning?

I plan on using ZFS (since our current plan is TrueNAS Core) running 8x 8-drive vDevs each in a RAID-Z1 configuration with each drive being 8TB. 

[Dutch_Master]

Given you have a 42U rack and need lots or storage but not much in computing power, consider used NetApp disk-shelves with a modest front-end server. Reserve space for more head-units if a High-Availability cluster is a future option. The new-ish EPYC-on-AM5 chips (EPYC 4000 series) are a good candidate for this scenario.

 

Here's a few vid's for inspiration:

 

HTH!

 

[Electronics Wizardy]

8 minutes ago, Crcoli737 said:

I plan on using ZFS (since our current plan is TrueNAS Core) running 8x 8-drive vDevs each in a RAID-Z1 configuration with each drive being 8TB. 

Why 8tb drives? I'd really suggest going with bigger drives like 24TB. Then you can get a much cheaper server model with less HDD bays. your also saving a lot in power usage.

 

 

[Dutch_Master]

10 minutes ago, Electronics Wizardy said:

bigger drives

Those are generally SMR, not a good idea in a cluster. I'm using 16TB CMR drives instead.

[Electronics Wizardy]

8 minutes ago, Dutch_Master said:

Those are generally SMR, not a good idea in a cluster. I'm using 16TB CMR drives instead.

I don't think there are any device managed SMR drives > 8TB out there currently. There are device managed SMR drives, but there pretty well labeled typically and not sold by major resellers, so I find it very unlikely you will buy one by accident. 

 

There are many models of 24TB drives that are CMR currently for sale.

[manikyath]

1 hour ago, Crcoli737 said:

for a local photography business

buy an off the shelf storage server solution.

 

some kind of rackmount server full of drive bays, and shove it full of high capacity drives. for example supermicro has a model that squeezes 36 drives into 4U with hotswap functionality (which is a must, because at this scale, drives popping is a given)

 

on that note, look into tape backups. for 600TB it makes a lot of sense.

[mariushm]

If you don't mind slightly higher power consumption (due to older generation hardware)  you can get servers that are pulled from datacenters.

 

For example Unixsurplus has a bunch of 4U servers that have up to 36 3.5" drive bays (24 in front, 12 in back) connected to SAS3/SATA3 backplanes. The motherboard also has 8 or 10 regular sata ports, so for example you could shove a bunch of sata enterprise SSDs inside  (if you can find power connectors for them) in those ports. 

 

$723 plus shipping (probably ~$100) :  https://unixsurplus.com/supermicro-4u-6048R-x10drh-server

 

Supermicro 4U 36 Bay SAS3 Server X10DRH-iT FreeNas with Configuration Options

Configuration * 2x E5-2683 V3 14C | 32GB DDR4 | Includes Supermicro Original Rails

1 PCI-E 3.0 x16 and 6 PCI-E 3.0 x8 expansion slots. (slot 1 & 2 occupied by controller and JBOD expansion port)
Dual 10GBase-T LAN w/ Intel® X540 ensures lightning-fast connectivity.
It also features 36x 3.5" hot-swap drive bays (24 front + 12 rear) with 32x 3.5" hot-swap SAS3/SATA3 drive bays and 4x NVMe hybrid drive bays (SAS3/SATA3/NVMe*).
The server uses SAS3 via Broadcom 3008 controller and IT mode and has 2x 2.5" optional hot-swap SATA drive bays (rear) and 4x 2.5" internal fixed drive bays.
It also supports IPMI 2.0/KVM over LAN/Media over LAN for remote management. Lastly, there are 7x 80mm middle cooling fans and 1280W redundant power supplies platinum level (94%).

 

Pre-built version of this config but with 128 GB DDR4 here for $1049 plus shipping : https://unixsurplus.com/Supermicro-Server-4U-36-Bay-X10drh-iT-Xeon-128Gb

 

Newer hardware, from around $1200 

 

ex $1250 plus shipping :  https://unixsurplus.com/supermicro-6049p-x11dph-t/

 

Supermicro 4U 36 Bay LFF Server Storage SSG-6049P-E1CR36L X11DPH-T with Configuration Options
Configuration * 2x Intel Xeon 6130 18C | 128GB DDR4 | Controllers: 1x 1x AOC-S3008L-L8e HBA 12Gb/s, 
1x AOM-SAS3-8i8E-LP Installed for external JBOD, 1x Intel X520-da2 dual SFP+ 10GB

 

Here's an older version that uses DDR3 registered ram, and SAS2 backplanes. It's cheaper and you get more ram but slower sas2 backplanes (though you said raw speed is not that big of a deal) but the downside is higher idle power consumption (ex 150-200w idle without any drives) 

 

$570 plus shipping  : https://unixsurplus.com/supermicro-847E16-X9DRH-ITF

 

Configuration * 2x E5-2690 V2 10C | 256GB DDR3

Supports up to 1TB ECC DDR3 with a speed of up to 1866MHz.
It has 16 DIMM slots for expansion, including 1 PCI-E 3.0 x16 and 6 PCI-E 3.0 x8
Intel® X540 Dual port 10GBase-T
8x SATA2 and 2x SATA3 ports
It comes with Integrated IPMI 2.0 and KVM, along with a dedicated LAN

Motherboard	:X9dRH-iTF * Integrated IPMI 2.0 Management
Server Chassis/ Case	:CSE-847E16-R1400LPB
Backplane	:2x Backplane:
*BPN-SAS2-846EL1 24-port 4U SAS2 6Gbps single-expander backplane
*BPN-SAS2-826EL1 12-port 2U SAS2 6Gbps single-expander backplane
PCI-Expansions slots	:Low Profile 4 x16 PCI-E 3.0, 1 x8 PCI-E 3.0, 1 x4 PCI-E 3.0 (in x8)
Caddies Included	:36x 3.5" Supermicro caddy
PSU Slots	:Dual
Power Supply	:2x 1400Watt Power Supply Gold
Rail Kit	:Supermicro Rev B Rail Kit 4U
Warranty	:60 Day Seller Warranty

 

 

You can also get  JBOD (just bunch of drives) enclosures  and connect them to the main server using an external SAS controller.

 

For example: 

 

$581 + $90 shipping :  Supermicro 45 Bay JBOD Expansion Server Shelf 847E16-RJBOD1 ALL Caddies w/ RAILS  : https://unixsurplus.com/supermicro-45-bay-jbod-expansion-server-shelf-847e16-rjbod1-all-caddies-w-rails/

 

45 3.5" drives, 24 caddies in front , 21 in the back,  dual 1400w psus (1100w each on 110v), sas2 ... a sas2 external controller is maybe $100, maybe $30 for the external cable to connect the enclosure to controller ... so you can simply connect this one to the main server and have room for expansion. 

 

Then it's just a matter of adding drives ... probably 16-18 TB drives would be the sweetspot (price vs capacity) right now ... but you could probably also shuck (remove the case of external drives) and temporarily put your external drives into the enclosure.

 

It may be worth investing in a tape drive, to dump projects and stuff that's not used for let's say more than one year, or for extra backup of very important projects.

 

Ex a ultrium-5 sas2 drive for around $1250 : https://www.amazon.co.uk/HP-LTO-5-Ultrium-Internal-Drive/dp/B008Y6VNLM/

 

1.5 TB (3 TB compressed) tapes are around $30 : https://www.amazon.co.uk/HP-C7975A-Ultrium-5-1-5TB-Tape/dp/B003F8MT5I/

 

 

- not saying Unixsurplus has the best deals but it's convenient website to show examples

 

ps.  you can get guaranteed CMR series  like WD Red Plus   (regular Red is SMR/CMR) 

 

red plus up to 14 TB (all CMR) : https://products.wdc.com/library/SpecSheet/ENG/product-brief-wd-red-plus-hdd.pdf

 

red pro up to 22 TB (all CMR) : https://products.wdc.com/library/SpecSheet/ENG/product-brief-wd-red-pro-hdd.pdf

 

wd gold up to 24 TB (all CMR) : https://products.wdc.com/library/SpecSheet/ENG/product-brief-wd-gold-hdd.pdf

 

wd purple (surveillance series) up to 8 TB (all CMR) : https://products.wdc.com/library/SpecSheet/ENG/product-brief-wd-purple-hdd.pdf

(supposedly firmware optimized for parallel writes and low latency to the detriment of read speeds)