Future-proofing Storage Server

[You_are_a_cunt]

So, my dad's electronics workshop is getting out of hand when it comes to storing its data. We're currently at 16 2TB drives with the 17th coming fairly soon (within a couple of months).

Currently I'm designing a system to easily access documentation on specific parts, appliances and so on, which isn't a problem in itself (it's fairly easy, actually) but I find myself in quite a pickle.

We're waiting on a HP Proliant DL380p Gen8 server that I managed to get fairly cheap as a barebone system (I've only had to purchase the memory for it, since it already came with the controllers and processors, two Xeon E5-2609v2), but the problem would only be sorted for a short while.

Since the server can only house a maximum of 25 drives (up to a total of 50TB of storage), it won't take long until storage will get out of hand once more.

 

Why is there so much data? Well, we keep documentation on every component we've ever used in the past 20-odd years, high res scans and photos of the appliances, client data, service manuals and so on.

We're at about 31TB of data right now with at least 12TB coming once we migrate all the hard files into digital (which is usually done on slow days when clients are scarce, so that might take a while).

Since we've spent about $3k on the Proliant, we're not keen on having to buy a second server next year to be able to have our databases easily accessible, yet we understand that it would likely need to happen at one point. If my estimation is at least accurate, we'd be past the 43TB marker by Q3 2016 (assuming the rest of the hard files are moved to digital by then) leaving the server with a "measly" 7TB of extra possible storage that would fill up by mid-2017.

 

Now, the issue I can't figure out is how to expand the overall storage capacity without breaking bank and giving some more storage room that would last for at least a couple of years before we need to invest in a second server.

This brings me onto a semi-brainfart idea I had at one point. Would it be possible to compress files (JPEGs and PDFs to be more precise) without losing quality? Pictures are 21MP and scans are roughly in the same area and most PDFs average out at about 70-80MB. Client data, fortunately, doesn't take up all that much storage since it's all in sql databases.

So, bottom of the line: I need to find a way to squeeze at least another 18TB of storage out of a server that has a maximum capacity of 50TB (so, 68TB in total, spread across 34 drives). For now we can use one of the PCs to give us another 12TB of storage (6x2TB) giving us a maximum of 62TB out of the 80TB that we figure would be enough for the next two years.

[Streetguru]

So, my dad's electronics workshop is getting out of hand when it comes to storing its data. We're currently at 16 2TB drives with the 17th coming fairly soon (within a couple of months).

Well you're always going to lose some quality when compressing images

 

but why don't you just put the really old stuff on either blu ray discs or like the 8TB archival drives and store them in a nice well packed safe? Or do you need access to stuff readily?

[jcw150]

Are you using RAID or any equivalent?

[multifrag]

Download Advanced JPEG Compressor and it will show you how much you can compress it until you will see a difference. Running uncompressed data is a bad optimization problem. Most of my rendered films, pictures go through 3 stages. Uncompressed finished project. Compressed archive. Deleted rendered images, but the save file preserved. So when I compress an image it can go from 100mb per image to just 500kb or even less

[You_are_a_cunt]

Well you're always going to lose some quality when compressing images

 

but why don't you just put the really old stuff on either blu ray discs or like the 8TB archival drives and store them in a nice well packed safe? Or do you need access to stuff readily?

Except for the client data everything needs to be readily available because you can't predict what a client might bring in for servicing. Even if I spend a couple of month sifting through ancient datasheets that aren't likely to ever be needed more than once in a blue moon, it still won't clear out more than 2-3TB of data.

Upgrading to 8TB drives would give us 96TB of storage space but that would mean having to migrate everything from the 2TB drives. While it would take a while, it might actually be the most cost-efficient solution, if I think about it. With the cost of the server and 12x8TB drives we'd be exceeding our budget by a fair margin (drives alone would cost in excess of $8k).

I'm hoping for something cheaper to implement on the short run

[You_are_a_cunt]

Are you using RAID or any equivalent?

Not planning on using RAID. Going with JBOD is possible, if not I'll be using RAID 0

[Streetguru]

Except for the client data everything needs to be readily available because you can't predict what a client might bring in for servicing. Even if I spend a couple of month sifting through ancient

With that amount of data i don't think anything is going to be cheap, aside from buying refurbished drives which probably isn't a good idea

 

the 8TB drives cost $200 each right now

 

https://pcpartpicker.com/part/seagate-internal-hard-drive-st8000as0002

I think you'd want to set it up so those drives turn off when not in use

 

if you want something that'll just run all day you'd want the HGST 4TB drives $159 each though

https://pcpartpicker.com/part/hitachi-internal-hard-drive-0s03664

[You_are_a_cunt]

Download Advanced JPEG Compressor and it will show you how much you can compress it until you will see a difference. Running uncompressed data is a bad optimization problem. Most of my rendered films, pictures go through 3 stages. Uncompressed finished project. Compressed archive. Deleted rendered images, but the save file preserved. So when I compress an image it can go from 100mb per image to just 500kb or even less

This could actually help. Even with a 50% disk size reduction it would clear up significant storage space

[You_are_a_cunt]

With that amount of data i don't think anything is going to be cheap, aside from buying refurbished drives which probably isn't a good idea

 

the 8TB drives cost $200 each right now

 

https://pcpartpicker.com/part/seagate-internal-hard-drive-st8000as0002

The cheapest 8TB SAS drive I can find here is $700 >.<

 

LE: 8TB SATA would be around $300

[Streetguru]

The cheapest 8TB SAS drive I can find here is $700 >.<

Well ya if you SAS you gotta pay, but you probably don't need it too badly, you have all your stuff backed up online to somewhere as well right?

[You_are_a_cunt]

Well ya if you SAS you gotta pay, but you probably don't need it too badly, you have all your stuff backed up online to somewhere as well right?

Currently nothing is backed up but we're in talks with a company to get a deal for off-site backup

[Streetguru]

Currently nothing is backed up but we're in talks with a company to get a deal for off-site backup

Well do that before anything else, and then you should be good to go with the 8tb archival SATA drives to back up the older data anyways. they should be fine overall.

[You_are_a_cunt]

Well do that before anything else, and then you should be good to go with the 8tb archival SATA drives to back up the older data anyways. they should be fine overall.

Can't do much until the server arrives and since it was bought from outside the country it's going to take a while for it to get here.

 

Photo compression alongside the 12 8TB drives should give us enough storage for the foreseeable future

[jcw150]

Not planning on using RAID. Going with JBOD is possible, if not I'll be using RAID 0

So no redundancy incase of drive failure?

[You_are_a_cunt]

So no redundancy incase of drive failure?

off-site backups should provide enough redundancy

[Streetguru]

Can't do much until the server arrives and since it was bought from outside the country it's going to take a while for it to get here.

 

Photo compression alongside the 12 8TB drives should give us enough storage for the foreseeable future

I assume you mean you're going to expand to 12, as like 6 should be good for now

[You_are_a_cunt]

I assume you mean you're going to expand to 12, as like 6 should be good for now

Yes, currently going with 6 and expanding to 12 once funds become available

[xgn]

You could always buy bigger harddrives 8 TB exist now but not worth it 4TB ones are cheap.

[You_are_a_cunt]

You could always buy bigger harddrives 8 TB exist now but not worth it 4TB ones are cheap.

The server controller cards seem to only support 2 and 8TB drives. Also, we got 2TB drives because the 4TB ones don't seem to work with our main PC or the docking station we've been using

[xgn]

The server controller cards seem to only support 2 and 8TB drives. Also, we got 2TB drives because the 4TB ones don't seem to work with our main PC or the docking station we've been using

Did you set them as GPT ? also Seagate archive v2 dosent 8 TB dosent cost too much. If its in windows server maybe you need to activate some driver policies not sure.

[You_are_a_cunt]

Did you set them as GPT ? also Seagate archive v2 dosent 8 TB dosent cost too much. If its in windows server maybe you need to activate some driver policies not sure.

that PC is pretty old and the storage controllers are pretty crappy. Now, with the new server inbound, I'm not worried about that PC anymore

[You_are_a_cunt]

Wooo, all parts are now ordered.

Server itself is due here next Thursday and the drives should be here either tomorrow or Monday.