Lifetime storage + Stuff?!?

[Verymanythings]

Hooowdy!

 

I'll star by giving you a quick intro to who i am, and what i do!

 

I'm a professional photographer, i take a crapton of photos and i record very many videos. I run a team of 3 people ( me + two others). 

 

To all our customers, we sell lifetime storage of ALL the photos and videos we make for them for a set rate, around 85$ one time payment. (usually wedding stuff + some product photography and family shoots). Nothing complicated.

 

Up until now i have been solving this by getting premium deals on usb flashdrives from a local tech shop i have ties to, but this is just excruciatingly painful, due to the fact that a single accident could wipe out everything. There's obvious problems with this, as you all understand.

 

NOW! I would like to solve this issue for myself, and i need your help to do it!

 

Below is some criterias for what i want.

 

A machine that can:

 

1. Store huge amounts of data (20-30 TB or more)

2. Double as a render PC which i or a member of my team can edit off when we're on the road.

3. Not break my bank, this is a part-time mission with limited amount of resources. (3000-4000$, defo not more than 5k)

4. Live forever.

 

Any help and suggestions is greatly appreciated!

 

Thanks for your time

[just tilt]

Just clarify your budget so we know what we're working with.

[Verymanythings]

3 minutes ago, just tilt said:

Just clarify your budget so we know what we're working with.

3000-4000$, defo not more than 5k)

[Enderman]

https://www.45drives.com/products/storinator-nas-av15-configurations.php

[Guest]

Could look at a QNAP TVS-671 with 6TB Red Pro drives for around $2.5-$3k US. Not 100% sure on the offloading render jobs though.

 

EDIT - you might also want to look at how you're backing that data up. I know Azure offers remote storage support for QNAP so you could use their cheap cold storage option.

[AshleyAshes]

3 hours ago, Verymanythings said:

To all our customers, we sell lifetime storage of ALL the photos and videos we make for them for a set rate, around 85$ one time payment. (usually wedding stuff + some product photography and family shoots). Nothing complicated.

 

Up until now i have been solving this by getting premium deals on usb flashdrives from a local tech shop i have ties to, but this is just excruciatingly painful, due to the fact that a single accident could wipe out everything. There's obvious problems with this, as you all understand.

Wait, let me get this straight... You offer your clients the indefinate preservation of any media you produce for them but then you only store a SINGLE instance of that data on a USB key?

 

...How fucking stupid can you be?  Like, I literally think that this might be FRAUD.

[Arika]

if you're a photography company that is offering "lifetime" storage, this is something you should already have....do your clients know you store their potentially irreplaceable data on a single flash drive?

 

EDIT: Miswording

[AshleyAshes]

9 minutes ago, Sierra Fox said:

if you're a photography company that is offering "lifetime" storage, this is something you should already have....do your clients know you store their potentially irreplaceable data on a single flash drive?

 

EDIT: Miswording

I don't even think OP is going to properly correct this.  This thread is asking about build a better local storage system and as well built as it can be it is still a SINGLE location for the data.  The OP thinks the system should 'Live Forever' so they seem to be pretty ignorant about data presevration.  No matter how well you build the basket, you still have all your eggs in ONE basket.  One fire or the right kind of power surge or PSU failure or USER ERROR could ruin all that data.

 

The smart plan is to outsource that to something offsite, Glacier or who knows what.  Then use some kind of 'colder' for local access like MDISC Blurays in a well organized and sorted system.  So in MOST cases when the client calls up because they want additional copies off their wedding photos, you PROBABLY have an optical disc on hand with it.  From then onward you have additional versions elsewhere incase those have failed.

 

ONE instance of data, no matter HOW well stored, is vunlerable.  The only real security for data is MULTIPLE instances, so that if you discover that an instance has been compromised, you can copy one of the other instances to replace the instances.  This is especially true if you're offering this as a COMMERCIAL SERVICE.

[Arika]

7 minutes ago, AshleyAshes said:

-a big snip-

soooo, 3 flash drives per customer? one in the right pocket, one in the left, and one in a shipping container next door.

[Verymanythings]

Thanks for the personal insults and whatnot!

 

Anyways, the pictures i take already has 3 backups(a couple of big eHDD's and a cold storage pc), this is not an issue, there's no need for you to debate this. The issue is i need to get rid of the ~100 flash drives i have on my hands aswell as get a better way of accessing these images and videos without having to (FIGURATIVELY SPEAKING) dig through a mountain of flash drives or take a 1 hr drive to my summerplace where my cold pc is, or plug in 6 different external HDD's trying to find what i'm looking for.

 

Enough of that.

The 45drives option seems appealing, but very very costly, i was able to push the price just below 4k by dropping phone support and pre-installation.

Also, what is Unraid? I have heard it mentioned but never really had an explanation of it.

 

The NAS thing is pretty much a no-go. I need to be able to double it as a render offloading machine. 

 

Looking out for more suggestions!

[Pyramiden]

I don't know anything about "render offloading", but I don't get why you would want your storage computer to do this?

I would try to find a HP Microserver Gen8. It is confirmed to be able to use 6TB drives, but I don't think there is a reason it would not be able to use bigger drives.
Put a P222 or P420 Raid controller in it, or run Windows Storage spaces and put 4 10TB drives in it. Add a boot drive to the 5th SATA port, find a E3-1260L or 1265L on Ebay for a bit more power and you are good to go. With the P222 you could even add an external enclosure. Maybe upgrade the RAM, if you want to (it requires ECC UDIMM).
And you could use Windows Server or Windows 10 Pro as OS.
Or a more professional focussed towerserver from HP or Dell.

[Verymanythings]

1 hour ago, Pyramiden said:

I don't know anything about "render offloading", but I don't get why you would want your storage computer to do this?

I would try to find a HP Microserver Gen8. It is confirmed to be able to use 6TB drives, but I don't think there is a reason it would not be able to use bigger drives.
Put a P222 or P420 Raid controller in it, or run Windows Storage spaces and put 4 10TB drives in it. Add a boot drive to the 5th SATA port, find a E3-1260L or 1265L on Ebay for a bit more power and you are good to go. With the P222 you could even add an external enclosure. Maybe upgrade the RAM, if you want to (it requires ECC UDIMM).
And you could use Windows Server or Windows 10 Pro as OS.
Or a more professional focussed towerserver from HP or Dell.

What i mean with render offloading is that if i'm on the train and working with a project in Premiere or likewise, i need to be able to throw the project over to the server and render it there, instead of rendering it on my laptop and effectively locking it down for 30-40 minutes.

[Pyramiden]

1 minute ago, Verymanythings said:

What i mean with render offloading is that if i'm on the train and working with a project in Premiere or likewise, i need to be able to throw the project over to the server and render it there, instead of rendering it on my laptop and effectively locking it down for 30-40 minutes.

What I mean to say is that I have no experience doing that. Is it CPU or GPU rendering? And what kind of application or OS do you need for it? I guess with a tower server you can just throw in a beefy CPU or GPU for rendering. Microserver Gen8 only has one PCIE slot, and Gen10 has no iLO and a slow CPU which can't be exchanged, so then I would look for something bigger.

[Copie]

If it was me i would keep a local copy of the files in a redundant storage (hdd based software raid + secondary backup offsite) then keep a cold copy in AWS Glacier storage as a worse case solution.

 

4k should be enough for a redundant 40tb solution with the remaining budget will get you 30tb of glacier storage for at least a year.

[morrisonicole]

Have you looked into the Storinator Workstation? https://www.45drives.com/products/workstation/
It's essentially a PC that has a ton of storage, it writes to HDD or SSD’s, providing control of where your data lives. The SSDs provide fast access to your data until you are finished with it, and then it can be migrated to the HDDs for longer-term storage. 

[Thermite]

On 3/19/2018 at 6:07 AM, Verymanythings said:

What i mean with render offloading is that if i'm on the train and working with a project in Premiere or likewise, i need to be able to throw the project over to the server and render it there, instead of rendering it on my laptop and effectively locking it down for 30-40 minutes.

You need to get one of the storinators and use windows server. You can have windows handle the raid and for render offloading windows will probably be the most capable os for what you want to do. I don’t have much experience with render offloading but if any os would support it, it would be windows server especially for adobe products like premiere. 45 drives man this what you need. They also have amazing support they will help you if you need it. 

[Revamp]

Sounds to me like you basically just want an actual PC with a ton of HDD space...Grab you a 8700k, 1060 (don't know if you're gpu or cpu rendering, but if GPU you can upgrade), 32gb ram, and a nice ventilated case, and a bunch of HDD lol...

 

Then keep up with your current storage methods if you truly use 3 copies of backup.

[kirashi]

On 3/19/2018 at 12:22 AM, Verymanythings said:

--SNIP--

get a better way of accessing these images and videos without having to (FIGURATIVELY SPEAKING) dig through a mountain of flash drives

--SNIP--

 

The 45drives option seems appealing, but very very costly, i was able to push the price just below 4k by dropping phone support and pre-installation.

Also, what is Unraid? I have heard it mentioned but never really had an explanation of it.

The NAS thing is pretty much a no-go. I need to be able to double it as a render offloading machine. 

In this case, I'd 100% recommend investing in a 45drives machine, or another dedicated server machine with lots of hot swap drive bays. You're asking essentially for a $2000-$3000 rending server with hot storage attached to it for about half what it's going to cost in hardware, drives, and time setting it up. (This is not an insult to you; just stating a fact.)

 

You've got 2 choices:

1. Spend around $5000-$8,000 and do it right the first time (hardware in a storage chassis with enough HDDs and the time to set this up)
2. Spend between $2500-$5000 and plus a few thousand when (not if) you have a failure and need to shutdown business while restoring backups

I'm just throwing rough figures around here, but my point is I'd do it right the first time since this is storing media files that essentially make up the bread and butter of your business operations, even if it costs a bit more up front. I'd recommend looking into either building out a PC and attaching some fast NAS storage appliance to the network, or spending a bit more on a dedicated server chassis (such as one from Norco or SuperMicro) and putting a dual Xeon workstation into production for rendering.

 

On a side note, it's great that you have both on-site and off-site cold backups, but do you also have cloud backups should disaster strike at both locations at the same time?

[Razor Blade]

I can appreciate the OP wanting to take on the task of storage in house. The fact that the OP also recognized the terrible solution they've been using as...inadequate is proof of acknowledging the problem. But from what I gather, it would probably be better to have a service handle that storage task. Even if you had a great machine that can store more than any data you can throw at it you still should think about off site backups to protect against things like fire, flood, natural disasters, etc.

 

Sorry to not be much of help in the original questions but my opinion would be to take the money you would have spent on a giant server and shop around for a business account with a data storage firm. You'll have a lot more flexibility and would not have to worry about redundancy, backups, software licenses, hardware failures, or obsolescence of said hardware. I have no idea what sort of guarantee you give your customers for the lifetime storage thing but at least if you leave that data to a company that focuses in keeping the data safe and sound, you can focus on your pictures.