How to Lose 100TB of Data: A Guide to My $2000 Mistake

[Aero_db]

I originally posted on the Unraid forum but wanted to share with the LTT community.

 

This is the story of how I managed to fry all 12 of my HDDs in moment of stupidity. (5 min read)

 

To start, we should talk about my setup. To simplify, I'll post my old signature to cut to the chase. But in short, it was built up over time. Slowly adding drives as needed to satiate my data hording. Much of it was Plex media but I had setup things like a Pi-Hole, Nextcloud, personal VPN (for when I wanted to use public Wi-Fi on my phone in a safer manner). Much of this I think is pretty common in our community so my server wasn't really that out-of-the-ordinary. 

 

Config - Pre-incident

Unraid Plus

MB: ASRock X370 Taichi | CPU: AMD Ryzen 7 1700 | RAM: 32GB DDR4 | Cooler: Cooler Master MasterLiquid Lite 240 | Case: Thermaltake Core X9 | Cache: CORSAIR FORCE Series MP510 (960GB) | Parity: 2 X WD Red 10TB | Array: 3 X WD Red 8TB, 4 X Seagate 8TB, 2 X WD Red 10TB | GPU: XFX Radeon RX 560 | PSU: EVGA SuperNOVA G2, 80+ GOLD 650W

 

(Side note- never went full enterprise/rack mounted due to some confusion around backplanes and concerns over fan noise)

 

So this all started back when I resolved to add more streaming capability to the plex server. I simply wanted to be able to more comfortably encode/direct-stream more concurrent streams. Toying with the Idea of adding a Nvidia GPU rather than my Radeon which doesn't support hardware acceleration. But thanks to some great advice on the unraid discord, I would only add 3 or so streams, not a meaningful improvement for the cost. So was still debating a full motherboard and CPU upgrade. But one thing was clear, I would likely need to upgrade my power supply to support any major changes, GPU, CPU or otherwise.

 

Christmas came and my wife, the incredible Rockstar that she is, blew passed our gift giving budget (for each other) to buy me a Seasonic 1300 gold. (before anyone says it, platinum or titanium would be nice, but the power savings would be nominal and the return on investment was negligible especially at the premium). Needless to say, I was elated. 

 

I planned to turn down the server and and carry out the upgrade in one day. If you have ever upgraded a PSU, its not that big a deal usually. 

 

For anyone who doesn't know. Power supply units convert my American 120v AC power into DC and deliver multiple voltages to the various components. The connectors on the PSU side of the power cables are often keyed or shaped in a unique manner to ensure that they only fit/plug into the correct voltage ports. Square peg, round hole type of thing.

 

Skilled hardware builders out there, you know what's coming. 

 

I checked the preexisting power cables. The motherboard and GPU cables didn't fit. Easy enough, new unit has cables. I swapped those out. I double checked my work, plugged it all in, flipped the switch to turn on and the PSU clicked. A Common sign of a power fault or fault protection. Something was wrong. Perhaps something wasn't fully connected or maybe a new cable was faulty. After all, even Seasonic for all their track record, isn't infallible. But nothing looks wrong and I don't NEED the new PSU today, so I'll reinstall the old unit PSU and have the new PSU exchanged as faulty and try again in a week or two. No big. 

 

Old PSU reinstalled, turned on, BIOS shows no SATA disks....started trouble shooting and checking. Pulled HDD and attempted to connect directly to a windows machine to just see if the device is detected.... nothing. None of them. 

 

If you didn't connect the dots to what has transpired. Though a square peg doesn't fit into a round hole, many other shapes do fit into a square. Including the wrong voltages being unintentionally sent to my disks, frying the circuitry to all 12 disks including the parity drives, not that 2 of those could repair this level of dumassary. 

 

I can only explain the emotion I felt as mourning the loss of a loved one. I mean, you must understand, this was my biggest hobby and pleasure. Learning the endless capabilities of what I could do with Unraid. 

 

I did check with well known data recovery services but with the sheer volume of data... the quote was $9,000-21,000 dollars. Though they did say that if no data was recovered, It would cost me zero. A testament to the undoubtable quality of work they would be able to offer. But this isn't enterprise data. the financial risk to me was nil. For more reasons that the obvious, this was not a viable path forward. 

 

I resolved to rebuild. The sweet spot for storage size, cost, and physical drive slots for me was going with 7 new 14tb drives. It would yield only 70tb of usable storage (rather than my previous 86tb) but with drives of this size, I was definitely going to need/want 2 parity disks again. Also that would help with the prior issue I had of only being able to fit 12 disks in my case. Now I would have 5 more slots to grow into. All in all the new disks would cost me nearly $1900 (pardon the dramatic title but its nearly 2K and the same goes for my 86tb of usable space out of the 100tb worth of drives).

 

Plus with the advent of things like Unmaniac, I will be able to reduce my storage needs considerably, re-encoding as I regrow my data. Shout out to the great Youtube creators teaching me. Linus tech tips, Spaceinvador One, the OGs. The new ones too, Capt. Chaz, The Mysticle, WhiteWitch Craft. I suppose I even owe a bit of thanks to bite my bits, garage escapades and all. 

 

The lesson learned that I must have forgotten since my computer building classes in high school...

 

Swap the power cables when swapping the PSU unless you're more than certain they are the same.

 

So that's where I am today. Preclearing the new 7 disks and soon I'll start installing dockers and seeing what I can get going. I tried to tell the story as best I could but feel free to ask anything you like that I may have left out. 

 

Config - Post-incident

unRAID Plus

MB: ASRock X370 Taichi | CPU: AMD Ryzen 7 1700 | RAM: 32GB DDR4 | Cooler: Cooler Master MasterLiquid Lite 240 | Case: Thermaltake Core X9 | Cache: CORSAIR FORCE Series MP510 (960GB) | Parity: 2 X WD Red 14TB | Array: 5 X WD Red 14TB, 2 X Sandisk SSD 240GB | GPU: XFX Radeon RX 560 | PSU: Seasonic 80Plus Gold 1300W

[Quackers101]

while this can be pretty bad, so long they are not badly fried and such. Maybe one can replace the motherboard inside of the HDD's?

A bit unsure, some might have other things that make the repair much longer and way more expensive?

Hopefully those are drives that could be repaired?

 

And that is quite a bit of TB you got there.

[SpookyCitrus]

That's unfortunate, and is a very common problem. My advice is to always, when swapping a PSU use the cables it came with. Even if you think or know they're the same or compatible cables. Doesn't matter to me, the extra few minutes you save in cable management isn't worth the cost of frying the new unit or components you have hooked to it if they don't work. I see this all the time at my shop it's just as common as the people who think that any Intel or AMD board will work with any Intel or AMD CPU.

[Enderman]

Mistake 1 was not having an off-site data backup, or at least a secondary place even if it's just hard drives on a shelf.

 

Mistake 2 was plugging your data into a computer you hadn't boot tested yet

 

Mistake 3 was using the wrong PSU cables or plugging them into the wrong ports

 

Very sad tho  sometimes you can take the PCB from another identical HDD and swap it and you can get the data off yourself, maybe look into those DIY data recovery videos.

If all your drives are the same maybe you'll only need to buy one new one and you can use the same PCB for each drive to get the data off.

[Kilrah]

Long story to hide the "don't reuse PSU cables" lesson, but... yeah, it sucks.

  

2 minutes ago, Enderman said:

If all your drives are the same maybe you'll only need to buy one new one and you can use the same PCB for each drive to get the data off.

That won't work since he has a drive array, he needs most of them online at the same time.

[The_russian]

All I can say is at least it was personal data, not data used for work. In the end it’s up to the consumer to know modular PSU cables are not made to the same pin out, especially from different manufacturers, but I would like to see a warning somewhere on the product or somewhere else that will be seen (not buried on page 35 section 2 of an owners manual) about it. 
 

Side note:

12 minutes ago, Aero_db said:

(5 min read)

Thank you for this, it is really appreciated when longer posts/blogs/articles include a read time.

[mariushm]

If you just shoved 12v on the 5v input, there's a decent chance you can repair the drives.

Often there's a tiny component by the power input which acts as a fuse and blows up... you can simply desolder that component and replace it  or just not install anything (if it's a zener diode) or just have a blob of solder or a wire there to override it.

 

That's... 5 minutes for each drive with a soldering iron to fix.

 

Worst case scenario, 12v got to the actual chips blowing them up, in which case you could in theory buy some faulty drives and transplant the chips to your drives.

 

Someone like Louis Rossman could probably make you a deal and repair them all for a big discount, since you know the reason and most likely all drives will have same failure reason so once one is diagnosed repairing the others won't take a lot of diagnostic time.

Would probably cost you at least $3-500 if they don't have to buy other drives to harvest components from other drives.

 

ps can you take as clear as possible pictures of your circuit boards on the broken drives?  if they're the same, 1-2 pictures is enough.

[ScratchCat]

1 hour ago, Kilrah said:

That won't work since he has a drive array, he needs most of them online at the same time.

Theoretically he could image the drives one by one and mount the images. However that requires enough space for the minimum number of drive images as well as space to move the files to a new pool.

[thrasher_565]

ya i fried my storage drive once too. a molex can be plugged in backwards...

[Mark Kaine]

I didn't get it... how did you connect the "wrong" cable to the motherboard (if that's what happened)? 

 

I thought that was *impossible*? o.o

 

 

[Kilrah]

He reused the cables from the old PSU with the new PSU instead of changing them, and they were wired differently.

[SpookyCitrus]

19 minutes ago, Mark Kaine said:

I didn't get it... how did you connect the "wrong" cable to the motherboard (if that's what happened)? 

I thought that was *impossible*? o.o

OP used the cables from their old PSU with the new PSU to save time and the cables weren't correct(had a different layout), which caused the wrong voltages to be sent to the hardware connected frying them.

[LogicalDrm]

-> Moved to Storage Devices

[Aero_db]

1 hour ago, Mark Kaine said:

I didn't get it... how did you connect the "wrong" cable to the motherboard (if that's what happened)? 

 

I thought that was *impossible*? o.o

 

 

My wording can be confusing. The existing mobo and GPU cables didn't fit the new PSU. So the mobo and GPU cables were replaced with the cables that came with the new PSU. The existing SATA cables did fit the new PSU, they were left as is. The voltages/pin-out on the ports were not the same, frying the SATA disks circuitry.

[Archer42]

This very sad event shows that any single system, no matter how reliable and smart it is in theory, is prone to various failures like this. Be it human error, hardware failure (while unlikely, PSU can still blow up and kill everything by itself too, for example), software failure (bugs always exist), malware or some external events.

 

IMO - learn from your mistakes, do not rebuild the same configuration, do not bother with redundant arrays at all and make separate copies of data on physically separate devices. This way any single mistake will be extremely unlikely to kill everything (and yes, mistakes always can and will happen, human nature...).

2 completely separate usb hdd-s, for example, are generally much safer than any single array, no matter how theoretically reliable it is. Be it unraid, zfs, or whatever...

 

Also as others have said - hdd-s killed by applying 12v to 5v input can often be easily repairable. Depends on multiple things, but IMO it is at least worth a try since you are not going to pay for professional data recovery anyway. Sometimes it is as easy as shorting blown fuse...

[Mihle]

On 1/18/2021 at 8:57 PM, Kilrah said:

Long story to hide the "don't reuse PSU cables" lesson, but... yeah, it sucks.

  

That won't work since he has a drive array, he needs most of them online at the same time.

If he used JBOD as he probably did because Unraid and didn't do anything special, no he don't need to have most of them online at the same time.

If it was RAID he would have tho but that's not how must us Unraid.

With JBOD you should be able to put in one and one and get the files off them, but depending on how you have set it up it might be messy because I don't know how folder structures works then.

I have never done it myself but it should be possible as each file is stores on one disk and not spread on multiple in JBOD.

[Mihle]

I have a question, why did you get an PSU with that much watt?

[Aero_db]

3 hours ago, Mihle said:

I have a question, why did you get an PSU with that much watt?

I was doing the pre-work for future upgrades to increase potential plex streams. Initially that plan was to buy an RTX 2000 series card once they are available again, but upon additional research it wasn't the best path forward with all the NVENC encoding limitations. I think it was something like 3 new streams or something like that. Then at that point the next best option was upgrading the CPU. Which for me, I was still determining the best path forward but was/am still leaning toward a R3000 or R5000 series. Which would subsequently call for a new mobo with a higher chipset. At that point, more power was needed beyond the 650 EVGA I had. 

 

I saw the need for higher power as an option to consider more 'future proof' PSU options. After all, they are generally something that doesn't change as much over the years. baring the coming intel 12v thing, if that ever takes hold. So higher wattage than what i need now, longer warranty type considerations started to play a big role. I have only heard good things about seasonic. it seemed to be a solid offering and there's always the general tone of a company that offers a 12 year warranty on their products. 

 

Also for those who do the digging, using a PSU generally below 75% of is capability, will find more power efficient AC to DC conversion, less heat dissipation and overall lower power usage excluding the core component needs.

 

so there was no immediate need for 1300 watts, it was absolutely overkill. but I also think that I could grow into it as things like RTX 3000/4000 and R6000(or what ever they call it) comes to be more attainable and/necessary. 

[Mihle]

2 hours ago, Aero_db said:

I was doing the pre-work for future upgrades to increase potential plex streams. Initially that plan was to buy an RTX 2000 series card once they are available again, but upon additional research it wasn't the best path forward with all the NVENC encoding limitations. I think it was something like 3 new streams or something like that. Then at that point the next best option was upgrading the CPU. Which for me, I was still determining the best path forward but was/am still leaning toward a R3000 or R5000 series. Which would subsequently call for a new mobo with a higher chipset. At that point, more power was needed beyond the 650 EVGA I had. 

 

I saw the need for higher power as an option to consider more 'future proof' PSU options. After all, they are generally something that doesn't change as much over the years. baring the coming intel 12v thing, if that ever takes hold. So higher wattage than what i need now, longer warranty type considerations started to play a big role. I have only heard good things about seasonic. it seemed to be a solid offering and there's always the general tone of a company that offers a 12 year warranty on their products. 

 

Also for those who do the digging, using a PSU generally below 75% of is capability, will find more power efficient AC to DC conversion, less heat dissipation and overall lower power usage excluding the core component needs.

 

so there was no immediate need for 1300 watts, it was absolutely overkill. but I also think that I could grow into it as things like RTX 3000/4000 and R6000(or what ever they call it) comes to be more attainable and/necessary. 

What you day about PSU and power efficiency only really matters for full load if you think it will actually spend a lot of time at full load. PSUs also get less efficent at low loads but then it's also drawing less. (5% of 800 is more than 5% of 200 % )

 

I doubt you would actually need 1300w ever unless you go dual CPU or GPU.

 

But I had not thought of warranty or how long it lasts. I don't know if other PSUs have as long warranty. That had not crossed my mind.

[wanderingfool2]

On 1/18/2021 at 11:56 AM, Enderman said:

Mistake 1 was not having an off-site data backup, or at least a secondary place even if it's just hard drives on a shelf.

 

Mistake 2 was plugging your data into a computer you hadn't boot tested yet

 

Mistake 3 was using the wrong PSU cables or plugging them into the wrong ports

 

Very sad tho  sometimes you can take the PCB from another identical HDD and swap it and you can get the data off yourself, maybe look into those DIY data recovery videos.

If all your drives are the same maybe you'll only need to buy one new one and you can use the same PCB for each drive to get the data off.

"Mistake 2" wouldn't have helped...the real mistake was mistake 3.

 

Actually with Mistake 1, people always say you need backups and should have offline backups...but I've always been curious if anyone actually thinks in regards to budgets of people...even at 86tb of storage assuming 12TB discs and if you were lucky to get it for $200 each...which is unrealistically cheap, that's 8 discs at $1600...for a cold storage backup.  If you had 86 tb of data, would you be willing to pay an extra $1600 for it to be safe in a cold storage?  Off-site backups make less sense for a consumer if they have 86tb of data...you're going to be paying a crazy amount per month.  It's only a mistake if there is critical data on the drives or data that you cannot live without.

 

 

[Kilrah]

6 minutes ago, wanderingfool2 said:

I've always been curious if anyone actually thinks in regards to budgets of people...even at 86tb of storage assuming 12TB discs and if you were lucky to get it for $200 each...which is unrealistically cheap, that's 8 discs at $1600...for a cold storage backup.  If you had 86 tb of data, would you be willing to pay an extra $1600 for it to be safe in a cold storage? 

Of course people think of budget... and it's up to anyone to decide what their data is worth to them, but a proper, reliable set of backups that covers you against almost anything does include offline and offsite.

 

Always best to suggest the reliable way, afterwards if the person considers it's too expensive/not worth it for them they can always scale down.

 

I have $2500 in backup drives and I consider it peanuts knowing this provides me with 2 full backup sets of everything I've done on a computer in the past 25 years.

[wanderingfool2]

7 minutes ago, Kilrah said:

Of course people think of budget... and it's up to anyone to decide what their data is worth to them, but a proper, reliable set of backups that covers you against almost anything does include offline and offsite.

 

Always best to suggest the reliable way, afterwards if the person considers it's too expensive/not worth it for them they can always scale down.

 

I have $2500 in backup drives and I consider it peanuts knowing this provides me with 2 full backup sets of everything I've done on a computer in the past 25 years.

That is where I disagree...it's not always best to "reliable way".  It's should be balancing the budget with the risk of failure/importance of data.  The part that I take a bit of an offense to is that Enderman called it a "mistake".  The mistake was not switching the cables with the new one.

 

For myself, it is about presenting the options and knowing the pitfalls of the options; but having someone choose one over the other doesn't make it a mistake (if things go wrong)...since you can effectively get almost double the capacity for the same "cost" but you risk losing the data not stored elsewhere.

[Enderman]

24 minutes ago, wanderingfool2 said:

"Mistake 2" wouldn't have helped...the real mistake was mistake 3.

 

Actually with Mistake 1, people always say you need backups and should have offline backups...but I've always been curious if anyone actually thinks in regards to budgets of people...even at 86tb of storage assuming 12TB discs and if you were lucky to get it for $200 each...which is unrealistically cheap, that's 8 discs at $1600...for a cold storage backup.  If you had 86 tb of data, would you be willing to pay an extra $1600 for it to be safe in a cold storage?  Off-site backups make less sense for a consumer if they have 86tb of data...you're going to be paying a crazy amount per month.  It's only a mistake if there is critical data on the drives or data that you cannot live without.

 

 

I don't see what the need for keeping 86TB of data is anyways.

If I had that much data that would mean I'm either a hoarder or was running a datacenter or something.

 

In the first case, you can just get rid of stuff since it's not important, or compress videos, or something.

In the second case it's probably a real job in which case $1600 is nothing since you make that back through the work you do.

[Kilrah]

10 minutes ago, wanderingfool2 said:

It's should be balancing the budget with the risk of failure/importance of data. 

And we can't know how important that data is unless the person mentions it. But if someone spends $3000ish for a 86TB storage server it's pretty unlikely they did that to only store worthless junk on it. Conversely if they could afford to spend $3k to mostly store worthless junk they probably are able to afford a backup as well for what matters.

[wanderingfool2]

19 minutes ago, Kilrah said:

And we can't know how important that data is unless the person mentions it. But if someone spends $3000ish for a 86TB storage server it's pretty unlikely they did that to only store worthless junk on it. Conversely if they could afford to spend $3k to mostly store worthless junk they probably are able to afford a backup as well for what matters.

That is sort of the point, if you don't know how whether the data is important or not one shouldn't jump to the conclusion that it is important enough to be backed up.

 

I have about 5TB of worthless data currently stored up, that I would miss if it got destroyed...but don't care enough about to have a backup of.  That data currently resides on a single drive that and isn't part of my backup cycle.  The point is, each person has different values on data and it doesn't help by saying you have to always backup