I turn to the LTT-forum to seek some advice on how to handle storage needs that goes beyond what normal consumers usually face.
I work on a project in which we expect to produce several PBs of data (initially 2-10 PB, but maybe eventually upwards of 50 PB, not counting data backup/redundancy). The data will be a mix of very large files (250+ gb) and millions of small files. The files will be generated on cameras equipped with a 10GbE ethernet port and fast flash memory, which will need to be unloaded into our storage solution. The data will be processed in parallel with new data being generated, so a high read/write speed is necessary.
My question is, if anyone here has any suggestions on where to look for solutions, either bought as a complete system or custom built. I'm aware of a few companies that deliver PetaByte-solutions, but I may be overlooking an obvious solution to my problem.
Furthermore, what kind of disk scheme would you guys suggest? A raid-based system, such as Raid-5/6/10, JBOD or something else?
Thanks for your input!