Moving 200TB of Data quickly?

[voyager_]

1 minute ago, Agall said:

 

 

Most executives seem like they'd rather run something till it dies than proactively prevent downtime with proactive replacement. Maybe if there's a technically educated executive who would otherwise know better.

 

I'm not in sales, but I have seen two different groups of senior executives as a civilian with the same role who both followed the same characteristics.

I've had good luck when getting them to do some math themselves, most management can understand labor hours and lost business revenue for being down. So ((average labor hour) x (number of down hours)) + (potential lost revenue) vs cost of device/software. Then tell them how it will further benefit the organization down the road. I'm having to do this in a meeting coming up to get a couple of software licenses that can do 50% of the work in 20% of the time. Techy sort of talk gets you nowhere 

[leadeater]

22 minutes ago, Agall said:

The problem with those solutions is cost to benefit, along with the higher upfront cost. Its also still putting your eggs in one basket too, where its susceptible to other types of failures compared to a separate redundant unit.

I'm not saying don't have another unit but having another isn't really as much extra protection as you might be thinking. Unless you have a good system of both keeping the data in sync real time and an automated or quick way to switch across that is practiced and understood by multiple people then you're throwing money at a solution that doesn't actually offer the benefits sought after.

 

So long as you have your RAID group properly laid out and using RAID-DP or RAID-TEC then disk failure is minimal if almost no concern, up until passing 8 year mark with the original disks still in use.

 

22 minutes ago, Agall said:

Even with redundancy within the chassis, you're still relying on a single layer that could otherwise be duplicated for about the same cost.

 

It is not the cost of another system and you are not relying on a single layer other than the physical disks, which again you have to be neglecting your maintenance and replacement life cycle to ever really be a factor.

 

Two Synology's is not better than one NetApp. Two Synology's also costs more than one NetApp 

 

But also like I mentioned QNAP dual controller ZFS is a cheaper option

[digitalscream]

Bit of a leftfield solution, but...most of that data isn't volatile - so it doesn't actually matter how long it takes to copy it to the alternate storage facility. Speed only matters at the point where you want to copy it back.

 

So...can someone like Backblaze help you here? Upload it all over the space of a month or two, then schedule in a day where you drive over to their facility and copy it back to your server using as many high-speed links as your server can support (10Gbe will take two days). Anything that's changed recently, you can just shove onto a couple of 18TB external drives (redundancy) and copy back half from each drive in parallel.

 

If they can do it for $2k or less, it'll be worth it. Any more than that, and it's up to management to decide what they'd rather do - pay $2k+ for a service, or $10k for redundant assets (if you can persuade them that "redundant" isn't a bad word in this context).

 

EDIT: Of course, if your current NAS isn't SSDs, then you have to factor in the fact that full-drive reads are also hard on the hardware, so you're reducing the lifespan by doing that (as you would if you were recovering the array from a dead drive). The opposite isn't necessarily an issue, since these aren't SSDs with masses of write cycles because it's effectively an archive array.

[Fatty 227]

17 hours ago, digitalscream said:

Bit of a leftfield solution, but...most of that data isn't volatile - so it doesn't actually matter how long it takes to copy it to the alternate storage facility. Speed only matters at the point where you want to copy it back.

 

So...can someone like Backblaze help you here? Upload it all over the space of a month or two, then schedule in a day where you drive over to their facility and copy it back to your server using as many high-speed links as your server can support (10Gbe will take two days). Anything that's changed recently, you can just shove onto a couple of 18TB external drives (redundancy) and copy back half from each drive in parallel.

 

If they can do it for $2k or less, it'll be worth it. Any more than that, and it's up to management to decide what they'd rather do - pay $2k+ for a service, or $10k for redundant assets (if you can persuade them that "redundant" isn't a bad word in this context).

 

EDIT: Of course, if your current NAS isn't SSDs, then you have to factor in the fact that full-drive reads are also hard on the hardware, so you're reducing the lifespan by doing that (as you would if you were recovering the array from a dead drive). The opposite isn't necessarily an issue, since these aren't SSDs with masses of write cycles because it's effectively an archive array.

I had no idea Backblaze would allow you to do something like that. Where are their data centers located?

[digitalscream]

50 minutes ago, Fatty 227 said:

I had no idea Backblaze would allow you to do something like that. Where are their data centers located?

I'm actually not sure, but...I know they allow you to send them an 18TB drive that they'll drop your data on and send back. What's the worst that can happen if you give them a call?

 

I'm sure there are other companies who provide similar services; you might have more luck with smaller companies who could do with a couple of grand for very little work. Got to be worth an hour of calling around?

[Fatty 227]

14 minutes ago, digitalscream said:

I'm actually not sure, but...I know they allow you to send them an 18TB drive that they'll drop your data on and send back. What's the worst that can happen if you give them a call?

 

I'm sure there are other companies who provide similar services; you might have more luck with smaller companies who could do with a couple of grand for very little work. Got to be worth an hour of calling around?

Not a bad idea!

[metadaddy]

On 3/12/2024 at 10:00 AM, Fatty 227 said:

I had no idea Backblaze would allow you to do something like that. Where are their data centers located?

[Full disclosure - I'm Pat Patterson, Chief Technical Evangelist at Backblaze].

 

Unfortunately, we don't allow you to do something like that. Our data centers are secure areas, and we don't let folks plug in their own machines there to transfer data.

 

What we do have is the Fireball - a 96 TB NAS that we can ship to new customers for them to plug in to their network, copy their data over, and ship back to us. We then plug it into our high speed network and copy it into one of their Backblaze B2 cloud object storage buckets. I don't think this is useful in this case, though, unless you want to keep a copy of your data in B2.

[digitalscream]

9 hours ago, metadaddy said:

[Full disclosure - I'm Pat Patterson, Chief Technical Evangelist at Backblaze].

 

Unfortunately, we don't allow you to do something like that. Our data centers are secure areas, and we don't let folks plug in their own machines there to transfer data.

 

What we do have is the Fireball - a 96 TB NAS that we can ship to new customers for them to plug in to their network, copy their data over, and ship back to us. We then plug it into our high speed network and copy it into one of their Backblaze B2 cloud object storage buckets. I don't think this is useful in this case, though, unless you want to keep a copy of your data in B2.

Ah, damn. Well, it was a bit of a Hail Mary idea

[aggie113]

With your compressed timeframe you may be left with using a 10GB sfp+ connection to another NAS with 22TB 7200rpm drives on a RAID 0.  Wouldn't use it for production but you might be willing to risk it for temp data storage.  Still would be 4-5k of drives alone.

[digitalscream]

7 minutes ago, aggie113 said:

With your compressed timeframe you may be left with using a 10GB sfp+ connection to another NAS with 22TB 7200rpm drives on a RAID 0.  Wouldn't use it for production but you might be willing to risk it for temp data storage.  Still would be 4-5k of drives alone.

At that point, he'd only need another two of those drives to make it RAID 6 for a bit more confidence and data-safety, at less than 10% additional cost.

 

And then, once he's got that far, he might as well just spend a bit more to relegate the old one to backup/hot spare duties, and move to the new one - thus halving the downtime. Or, more likely given management shenanigans, sell the old one and continue with minimal data safety

[aggie113]

2 minutes ago, digitalscream said:

At that point, he'd only need another two of those drives to make it RAID 6 for a bit more confidence and data-safety, at less than 10% additional cost.

 

And then, once he's got that far, he might as well just spend a bit more to relegate the old one to backup/hot spare duties, and move to the new one - thus halving the downtime. Or, more likely given management shenanigans, sell the old one and continue with minimal data safety

I'm assuming low cost for the 10-12bay NAS that would be holding the data for a short time.  Doesn't give him parity with the new NAS vs the old but is cheaper and thus might be management approved.  

[voyager_]

1 hour ago, aggie113 said:

With your compressed timeframe you may be left with using a 10GB sfp+ connection to another NAS with 22TB 7200rpm drives on a RAID 0.  Wouldn't use it for production but you might be willing to risk it for temp data storage.  Still would be 4-5k of drives alone.

Raid 0 should never be used in any sort of real world use case, way to much risk even if everything is currently backed up to tape. As @digitalscream mentioned for a small percentage more OP could atleast go raid 6. But factoring cost/time/ potential loss of business/revenue OP might as well put together a new NAS to transfer over everything then re-configure the current NAS and use it as a mirror or even re-sell if they really need the money. 

[Bdavis]

I would build a new system or buy a used system depending on budget and put trueNAS on it. Then use the old system as a backup. If one goes down you should be able to switch to the other one quickly. Tapes should be used on a schedule and kept off site.