PetaByte over USB?

[GrockleTD]

2 hours ago, Sayori said:

it'd cost around 600,000 USD, if you used these 2tb USB drives, which are the largest capacity I can find and are 1,399 each. 

you are implying that we were thinking of using flash drives instead of external usb hard drives.

[ScratchCat]

23 minutes ago, Grockle88 said:

i was trying to stick with semi-possible figures... im pretty sure those ssd's don't even exist yet.

Toshiba and Nimbus have shown working models, Samsung has a 30TB model currently available, 33 drives then.

18 minutes ago, M.Yurizaki said:

If you're asking them to replace something that they need to run reliably well, especially after a few incidents where Linus probably lost a few years of his life due to stress, I think mucking with their backup is the last thing they want to try something radical with.

 

A smaller scale thing on the side sure, but not as a serious upgrade to their existing solution.

Practicality is not the target here - this would be a topic for a Holy $hit episode. If the 100TB drives were available it would be reasonable to assume that Linus would tape them all together and call it a "1 Petabyte USB stick".

[GrockleTD]

2 hours ago, ScratchCat said:

Toshiba and Nimbus have shown working models, Samsung has a 30TB model currently available, 33 drives then.

Practicality is not the target here - this would be a topic for a Holy $hit episode. If the 100TB drives were available it would be reasonable to assume that Linus would tape them all together and call it a "1 Petabyte USB stick".

oh good lord i love it!

[GrockleTD]

2 hours ago, ScratchCat said:

Toshiba and Nimbus have shown working models, Samsung has a 30TB model currently available, 33 drives then.

Practicality is not the target here - this would be a topic for a Holy $hit episode. If the 100TB drives were available it would be reasonable to assume that Linus would tape them all together and call it a "1 Petabyte USB stick".

memes aside, i would legitamately love to see if this could be done.

[Adorable Cat]

2 minutes ago, Grockle88 said:

you are implying that we were thinking of using flash drives instead of external usb hard drives.

eh, it's only like a 500k dollar difference.

[GrockleTD]

2 hours ago, Sayori said:

eh, it's only like a 500k dollar difference.

lol ikr?

 

[GrockleTD]

and then the thread spontaneously dies...

[FinnJim3]

it is possible ; max usb devices on a single port is 127 the max HDD size at the moment is 12 TB 127x12 is 1524 so you could have 1.524 petabytes of usb storage. so its possible 

[GrockleTD]

4 hours ago, FinnJim3 said:

it is possibly ; max usb devices on a single port is 127 the max HDD size at the moment is 12 TB 127x12 is 1524 so you could have 1.524 petabytes of usb storage. so its possible 

and then there is the whole fact that there is never only 1 usb port on a computer/motherboard.

[FinnJim3]

2 hours ago, Grockle88 said:

and then there is the whole fact that there is never only 1 usb port on a computer/motherboard.

meaning that you could have 8PB of usb storage with 8 usb ports

[GrockleTD]

10 hours ago, FinnJim3 said:

meaning that you could have 8PB of usb storage with 8 usb ports

yesss...

 

[GrockleTD]

11 hours ago, FinnJim3 said:

meaning that you could have 8PB of usb storage with 8 usb ports

or you could have 4 PB in an RAID array

[Lensberger]

Revisiting this old thread, I really think this would be a realistic, yet fun project! In theory, it shouldn't be hard, but I'd really like to see whether it works in practice; and wouldn't it be amazing to have created the world's largest USB drive array!? 

 

Let's make it even more fun, and imagine the following:

* no dedicated multi-drive enclosures; just use good old external usb hard drives

* ... hanging off a Raspberry Pi, because why not?

 

84 x 12TB drives ... you get those easily, = 1008TB (e.g. Amazon: WD 12TB for £220)

Let's assume 4-port, externally powered USB hubs (since most multi port ones apparently are a chain of 4-port ones internally anyway, but that gives us less control)

So we take 21 of those hubs (i.e. 84 ports) ...

... and then 6 more hubs, with 24 ports in total ...

... and then 2 more hubs, to provide ports for those ...

... and those two hubs connect to your Raspberry Pi.

 

Now, the total device limit on USB is 128 devices. We'd have 84 drives, + 29 USB hubs ... so that sounds ok. Also, no more than 5 external stages in the daisy chain. Not a problem: from the Raspi USB port it's 2 "level 1" hubs, going to 6 "level 2" hubs, going to 21 "level 3" hubs, and then level 4 is the actual external HDDs.

 

AFAIK, Linux shouldn't have a problem handling 84 drives, either. -- So then bunch them all into a BTRFS pool (since ZFS doesn't run well on a Raspi, and for ZFS on a Petabyte array, you'd need *a lot* of RAM), and see what happens.

 

I imagine it will look like a Rat King. (https://en.wikipedia.org/wiki/Rat_king)

 

So, are we agreed that this is possible, and that we want to see it happen!? Or are we agreed? (There's only one answer, people!)

 

[X-System]

RAID controller card is always better for largest drive array. For example, ARECA ARC-1884ixl controller card supports up to 512 drives using SAS expander

 

[Lensberger]

On 6/3/2020 at 10:10 AM, X-System said:

RAID controller card is always better for largest drive array. For example, ARECA ARC-1884ixl controller card supports up to 512 drives using SAS expander

 

 

Of course this wasn't a serious suggestion for how to build a Petabyte array; rather, it's intended to be an insane, fun, and weird experiment, to see whether it can be done!