1.5 Petabype NAS on Mining-Mainboard possible?

[Felixfromgermany]

So to start, I just wanna say that this of course is a silly and inpractical idea, but the point is, that I think that it could be done.

 

I was browsing ebay recently and I fount an AsRock H110 BTC+ mining mainboard with 12 PCI-E x1 Ports (for only 30€!). I had an idea so I loocked up if there were PCI-E x1 to SATA adapters and sure enough there were some on amazon (like this one: Amazon-Link). If you were to buy 12 of them and plug them into the mainboard you would have (with the ports from the board) a total of 100 SATA ports. Each equipped with a IronWolf Pro (or similar) 16tb would make a raw capacity of 1.6 Petabyte. Throw in an old skylake i5 maybe and a cheap GPU for the one PCi-E 16x Slot and you would have a funktional NAS.

 

There would still be massive problems, like how to power 100 drives, but its only a thought experiment anyways.

Now I can't wait for you guys to tell me why this would never work.

 

 

 

(Please don't be too hard with my grammar, english is not my main language)

[Kilrah]

Work? Yes probably. Give the performance you'd expect? Not really. 

These cards are PCE2 x1, so about 500MB/s / card (that's the speed you can achieve with 2 IronWolfs, so all other 6 will be bottlenecked).

 

All the PCIex1 slots connect to the chipset, so you'll be bottlenecked to 4GB/s to the CPU at most (i.e. same as one PCIe3x4 NVMe drive) not counting any overhead for the parity drives etc, as well as any slowdowns from the array layout and CPU bottleneck.

 

That sounds fast, but accessing 1.6PB at 4GB/s would take almost 4 months... also 100 16TB IronWolfs will cost about $40'000. When you're putting that much in the drives you'd rather put 2-3k more on being able to access them efficiently.

[Windows7ge]

I can think of how there would be massive I/O bottlenecks assuming the system is stable. Not even sure how you'd RAID that all together. Then the power required to spin them up. During operation the Amps required isn't bad but initial spin up is really demanding with more disks and without Staggered Spin-up most normal PSU's would probably hit over-current protection limits and shutdown.

[Tristerin]

The two above me know 100x more than I, and the first thing I thought was....there are only so many PCIe lanes that can be used at a time!

[mariushm]

You can buy HBA cards for as low as 10$ each ... ex see these sata2 (300MB/s) gigabyte cards: link

Most are 8 port, but there's also 16 port or even 32 port ... here's example of 16 port and example with 32 ports

A lot of controller cards support port multipliers, which take one sata port and create 5 sata ports ... the 5 devices then "share" the original bandwidth of 300 / 600 MB/s (depending on version) ... here's a sata2 multiplier as example, for <15$ : link

 

You can also use pci-e risers and get them working in pci-e x1 slots, naturally you'd get less bandwidth, but they'd work. Though you don't really have to ... nowadays you can get a threadripper board with 7 or 8 slots and if you want to, you could further split the x16 slots into 2 x8 or 4x4 slots (boards support bifurcation)

 

So there are solutions which don't involve installing 10+ controllers because often when you go over 7-8 devices, you start to get into problems with drivers.

 

Some modern drives support suspend (reusing a pin on the 3.3v of sata connector) and some sata controllers support staggered spinup, so starting up 100 drives is not really an issue.

You'd just need multiple power supplies, or a dedicated power supply for 5v ..as a drive would use around 0.7a on average you'd need at least around 75A on 5v while most modern power supplies only deliver 20A max on 5v

[manikyath]

Just now, Tristerin said:

The two above me know 100x more than I, and the first thing I thought was....there are only so many PCIe lanes that can be used at a time!

its not actually more lanes than a normal board, the special thing about this board is that just about every lane the platform has available is split out into individual 1x slots.

 

the thing here is that so many drives pose complex issues, beyond just the headache of powering them.

 

also, with this many high-capacity drives, the cost of a server to shove them into really is just relative...

[Tristerin]

1 minute ago, manikyath said:

its not actually more lanes than a normal board, the special thing about this board is that just about every lane the platform has available is split out into individual 1x slots.

 

the thing here is that so many drives pose complex issues, beyond just the headache of powering them.

 

also, with this many high-capacity drives, the cost of a server to shove them into really is just relative...

That's what I meant - the CPU, and the chipset only have so many, what he was describing ran out of PCIe lanes very quickly lol (his amazon link is for a breakout board for even more SATA ports)

 

I use similar boards in my mining farm

[beersykins]

Technically you can do this with any board that supports bifurcation using breakouts.

 

I'd just get an enterprise data shelf at that point, more backplane capacity, more redundancy.  After a certain point you need to plan for larger failure domains instead of like 'lol 100 drive RAID6'