Request for advice on PB-storage solutions

[SimonRibergaard]

I turn to the LTT-forum to seek some advice on how to handle storage needs that goes beyond what normal consumers usually face. 

 

I work on a project in which we expect to produce several PBs of data (initially 2-10 PB, but maybe eventually upwards of 50 PB, not counting data backup/redundancy). The data will be a mix of very large files (250+ gb) and millions of small files. The files will be generated on cameras equipped with a 10GbE ethernet port and fast flash memory, which will need to be unloaded into our storage solution. The data will be processed in parallel with new data being generated, so a high read/write speed is necessary. 

 

My question is, if anyone here has any suggestions on where to look for solutions, either bought as a complete system or custom built. I'm aware of a few companies that deliver PetaByte-solutions, but I may be overlooking an obvious solution to my problem. 

 

Furthermore, what kind of disk scheme would you guys suggest? A raid-based system, such as Raid-5/6/10, JBOD or something else? 

Thanks for your input!

[samcool55]

5 minutes ago, SimonRibergaard said:

I turn to the LTT-forum to seek some advice on how to handle storage needs that goes beyond what normal consumers usually face. 

 

I work on a project in which we expect to produce several PBs of data (initially 2-10 PB, but maybe eventually upwards of 50 PB, not counting data backup/redundancy). The data will be a mix of very large files (250+ gb) and millions of small files. The files will be generated on cameras equipped with a 10GbE ethernet port and fast flash memory, which will need to be unloaded into our storage solution. The data will be processed in parallel with new data being generated, so a high read/write speed is necessary. 

 

My question is, if anyone here has any suggestions on where to look for solutions, either bought as a complete system or custom built. I'm aware of a few companies that deliver PetaByte-solutions, but I may be overlooking an obvious solution to my problem. 

 

Furthermore, what kind of disk scheme would you guys suggest? A raid-based system, such as Raid-5/6/10, JBOD or something else? 

Thanks for your input!

Honestly when we are talking about PB's, you should go with a professional company that can offer a complete package. Because apart from storage, the performance needs to be there and the network needs to handle everything as well.

 

Maybe it's a good idea to contact a company like 45drives and ask them for help. This looks like a very big task 1 person shouldn't do on its own.

[jj9987]

As said above, at that scale, you're better off with enterprise solutions that take into account your exact needs, usage, budget.

[SimonRibergaard]

Thanks for both of your quick replies!
 

I’m already talking to 45drives, but with an investment of this size, we will most likely have to put out a tender, so I want to make sure that I check out the different options available and that I don’t miss an obvious solution :)

 

Again, appreciate your input!

[Jarsky]

For that amount of data and at the rates you probably want to be unloading it, you should be looking at SAN solutions; i'd be looking at talking with companies like NetApp & Nimble to see what they can offer and what they recommend rather than going to a forum on the internet asking about RAID levels. There is *so* much more to worry about when you're looking at this level of enterprise data storage.  Also consider backup and monitoring solutions you need to put in place for all that storage. Veamm have excellent integration with both the above vendors appliances, and something like Grafana is great for agentless monitoring. Also consider the access to the storage as well, you probably want an Active Directory environment to do role based access, or at the very least an LDAP. 

[Nick7]

You need to be more specific with your needs. What you most likely will need is NAS (not block storage).

You mention 'high r/w speeds' - you need to be quite more specific - because at >PB size you'll often go also into NL (aka SATA) area with drives, since in the end it will be quite a lot of drives. If you really need extreme performance, you'll need to go SSD/NVMe way, which is a LOT more expensive, especially since you say you intend to store video, which is not compressible or possible to dedup.

Also, with workload you mention, tiering does not seem to be efficient.

You need to have more specific requirements for RFP and tender, and see from 'big boys' what they offer (like NetApp, EMC, HPE, ...).