Jump to content

Storage Server Crashed - Backup Server Introduction

nicklmg
 Share
Posted

Seagate 8 TB Enterprise HDD

Amazon: http://geni.us/4CSt
NCIX: http://bit.ly/1YD2sb2

 

Intel Xeon Quadcore

Amazon: http://geni.us/PDl

 

Kingston ECC DDR3 32 GB

Amazon: http://geni.us/1AL0

NCIX: http://bit.ly/1TiRoP2

 

iStarUSA Redundant PSU

Amazon: http://geni.us/SV7

NCIX: http://bit.ly/1OMCz3u

 

iStarUSA Toolless Case

Amazon: http://geni.us/3dU3
NCIX: http://bit.ly/1Opi5nb

 

Backups are important. We realize that now more than ever, as we experienced a RAID failure on one of our servers very recently...

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

why are there forumers more knowledgeable at planning out storage servers than an entire company

Why do they know as good or better than us, but then don't do things right anyways? :P

 

I guess that whole "do as I say, not as I do" continues to hold true here...

Solve your own audio issues  |  First Steps with RPi 3  |  Humidity & Condensation  |  Sleep & Hibernation  |  Overclocking RAM  |  Making Backups  |  Displays  |  4K / 8K / 16K / etc.  |  Do I need 80+ Platinum?

If you can read this you're using the wrong theme.  You can change it at the bottom.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Backup sever room's on fire...

OFF TOPIC: I suggest every poll from now on to have "**CK EA" option instead of "Other"

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Why do they know as good or better than us, but then don't do things right anyways? :P

 

I guess that whole "do as I say, not as I do" continues to hold true here...

I mean I know we should back up our data but I don't give a rat's ass as long as it isn't some random things I have.

m6Trx.png

Check out my guide on how to scan cover art here!

Local asshole and 6th generation console enthusiast.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Didn't Linus specifically use RAID 5 for redundancy and crash recovery?

Main Rig:-

Ryzen 7 3800X | Asus ROG Strix X570-F Gaming | 16GB Team Group Dark Pro 3600Mhz | Corsair MP600 1TB PCIe Gen 4 | Sapphire 5700 XT Pulse | Corsair H115i Platinum | WD Black 1TB | WD Green 4TB | EVGA SuperNOVA G3 650W | Asus TUF GT501 | Samsung C27HG70 1440p 144hz HDR FreeSync 2 | Ubuntu 20.04.2 LTS |

 

Server:-

Intel NUC running Server 2019 + Synology DSM218+ with 2 x 4TB Toshiba NAS Ready HDDs (RAID0)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Did you mean to fill out those amazon/NCIX links and just forget? Or am I missing something here?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

why are there forumers more knowledgeable at planning out storage servers than an entire company

 

Because the "entire company" is still pretty much just Linus (and video editors).  He's not very good at asking for advice it seems and ends up screwing up a lot.  Whole room water cooling...looking at you.

Workstation:  14700nonk || Asus Z790 ProArt Creator || MSI Gaming Trio 4090 Shunt || Crucial Pro Overclocking 32GB @ 5600 || Corsair AX1600i@240V || whole-house loop.

LANRig/GuestGamingBox: 9900nonK || Gigabyte Z390 Master || ASUS TUF 3090 650W shunt || Corsair SF600 || CPU+GPU watercooled 280 rad pull only || whole-house loop.

Server Router (Untangle): 13600k @ Stock || ASRock Z690 ITX || All 10Gbe || 2x8GB 3200 || PicoPSU 150W 24pin + AX1200i on CPU|| whole-house loop

Server Compute/Storage: 10850K @ 5.1Ghz || Gigabyte Z490 Ultra || EVGA FTW3 3090 1000W || LSI 9280i-24 port || 4TB Samsung 860 Evo, 5x10TB Seagate Enterprise Raid 6, 4x8TB Seagate Archive Backup ||  whole-house loop.

Laptop: HP Elitebook 840 G8 (Intel 1185G7) + 3080Ti Thunderbolt Dock, Razer Blade Stealth 13" 2017 (Intel 8550U)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Because the "entire company" is still pretty much just Linus (and video editors).  He's not very good at asking for advice it seems and ends up screwing up a lot.  Whole room water cooling...looking at you.

kek

Check out my guide on how to scan cover art here!

Local asshole and 6th generation console enthusiast.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Because the "entire company" is still pretty much just Linus (and video editors).  He's not very good at asking for advice it seems and ends up screwing up a lot.  Whole room water cooling...looking at you.

 

which is half the fun though :P  Love the fly by the seat of our pants style they've always had!

 

That said, it really sucks they had this happen.  Too bad they don't make "sorry your RAID array crashed" cards... I guess an imaginary "get well soon" will have to do. :)

Solve your own audio issues  |  First Steps with RPi 3  |  Humidity & Condensation  |  Sleep & Hibernation  |  Overclocking RAM  |  Making Backups  |  Displays  |  4K / 8K / 16K / etc.  |  Do I need 80+ Platinum?

If you can read this you're using the wrong theme.  You can change it at the bottom.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Serious reply, my assumption if my LSI card died would be I would pop in a new one and it would redetect the raid array and hum along happily.  What brand was it that failed?

Workstation:  14700nonk || Asus Z790 ProArt Creator || MSI Gaming Trio 4090 Shunt || Crucial Pro Overclocking 32GB @ 5600 || Corsair AX1600i@240V || whole-house loop.

LANRig/GuestGamingBox: 9900nonK || Gigabyte Z390 Master || ASUS TUF 3090 650W shunt || Corsair SF600 || CPU+GPU watercooled 280 rad pull only || whole-house loop.

Server Router (Untangle): 13600k @ Stock || ASRock Z690 ITX || All 10Gbe || 2x8GB 3200 || PicoPSU 150W 24pin + AX1200i on CPU|| whole-house loop

Server Compute/Storage: 10850K @ 5.1Ghz || Gigabyte Z490 Ultra || EVGA FTW3 3090 1000W || LSI 9280i-24 port || 4TB Samsung 860 Evo, 5x10TB Seagate Enterprise Raid 6, 4x8TB Seagate Archive Backup ||  whole-house loop.

Laptop: HP Elitebook 840 G8 (Intel 1185G7) + 3080Ti Thunderbolt Dock, Razer Blade Stealth 13" 2017 (Intel 8550U)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Serious reply, my assumption if my LSI card died would be I would pop in a new one and it would redetect the raid array and hum along happily.  What brand was it that failed?

well apparently (paraphrasing here, but) it damaged the integrity of the array as it died so that probably wouldn't work.

Solve your own audio issues  |  First Steps with RPi 3  |  Humidity & Condensation  |  Sleep & Hibernation  |  Overclocking RAM  |  Making Backups  |  Displays  |  4K / 8K / 16K / etc.  |  Do I need 80+ Platinum?

If you can read this you're using the wrong theme.  You can change it at the bottom.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

why are there forumers more knowledgeable at planning out storage servers than an entire company

Pretty sure there are forum members who have careers around planning and managing storage servers so... yeah that would probably be why.

Also, it's not like we're "an entire company" that creates storage servers for a living.

 

Didn't Linus specifically use RAID 5 for redundancy and crash recovery?

Doesn't help much when it's the RAID card itself that dies.

 

Did you mean to fill out those amazon/NCIX links and just forget? Or am I missing something here?

People bitch when it's not posted early enough, people bitch when it's posted without links... Can't win lol.

 

Because the "entire company" is still pretty much just Linus (and video editors).  He's not very good at asking for advice it seems and ends up screwing up a lot.  Whole room water cooling...looking at you.

There often isn't time to ask for advice when you have video deadlines to make. Trial and error is sometimes the only execution strategy that's feasible solely due to time constraints.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Pretty sure there are forum members who have careers around planning and managing storage servers so... yeah that would probably be why.

Also, it's not like we're "an entire company" that creates storage servers for a living.

 

Doesn't help much when it's the RAID card itself that dies.

 

People bitch when it's not posted early enough, people bitch when it's posted without links... Can't win lol.

 

There often isn't time to ask for advice when you have video deadlines to make. Trial and error is sometimes the only execution strategy that's feasible solely due to time constraints.

well i mean i know you're not designed to create entire datacenters but i mean it must've been a thought at least somewhere

Check out my guide on how to scan cover art here!

Local asshole and 6th generation console enthusiast.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Doesn't help much when it's the RAID card itself that dies.

Ouch, that's bad luck then. No amount of prep or planning could have avoided that.

Was it an intergrated card or an external one?

Main Rig:-

Ryzen 7 3800X | Asus ROG Strix X570-F Gaming | 16GB Team Group Dark Pro 3600Mhz | Corsair MP600 1TB PCIe Gen 4 | Sapphire 5700 XT Pulse | Corsair H115i Platinum | WD Black 1TB | WD Green 4TB | EVGA SuperNOVA G3 650W | Asus TUF GT501 | Samsung C27HG70 1440p 144hz HDR FreeSync 2 | Ubuntu 20.04.2 LTS |

 

Server:-

Intel NUC running Server 2019 + Synology DSM218+ with 2 x 4TB Toshiba NAS Ready HDDs (RAID0)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Ouch, that's bad luck then. No amount of prep or planning could have avoided that.

Was it an intergrated card or an external one?

 

When critical servers are designed, RAID card failure is planned for.  IT guys are anal about making sure that replacement cards are the same firmware revision and everything as the dead card to maximize the recovery chance.

 

Chipset raid wouldn't work well for SSD raid.  It also is limited to maybe 6-8 drives.

Workstation:  14700nonk || Asus Z790 ProArt Creator || MSI Gaming Trio 4090 Shunt || Crucial Pro Overclocking 32GB @ 5600 || Corsair AX1600i@240V || whole-house loop.

LANRig/GuestGamingBox: 9900nonK || Gigabyte Z390 Master || ASUS TUF 3090 650W shunt || Corsair SF600 || CPU+GPU watercooled 280 rad pull only || whole-house loop.

Server Router (Untangle): 13600k @ Stock || ASRock Z690 ITX || All 10Gbe || 2x8GB 3200 || PicoPSU 150W 24pin + AX1200i on CPU|| whole-house loop

Server Compute/Storage: 10850K @ 5.1Ghz || Gigabyte Z490 Ultra || EVGA FTW3 3090 1000W || LSI 9280i-24 port || 4TB Samsung 860 Evo, 5x10TB Seagate Enterprise Raid 6, 4x8TB Seagate Archive Backup ||  whole-house loop.

Laptop: HP Elitebook 840 G8 (Intel 1185G7) + 3080Ti Thunderbolt Dock, Razer Blade Stealth 13" 2017 (Intel 8550U)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

When critical servers are designed, RAID card failure is planned for.  IT guys are anal about making sure that replacement cards are the same firmware revision and everything as the dead card to maximize the recovery chance.

Sure, you can plan for the replacement but no amount of planning could have prevented the card from dying. And that was my thinking, if it was external then they coulda replaced it like for like, chances are they wouldn't have needed to even rebuild the array after the swap.

Main Rig:-

Ryzen 7 3800X | Asus ROG Strix X570-F Gaming | 16GB Team Group Dark Pro 3600Mhz | Corsair MP600 1TB PCIe Gen 4 | Sapphire 5700 XT Pulse | Corsair H115i Platinum | WD Black 1TB | WD Green 4TB | EVGA SuperNOVA G3 650W | Asus TUF GT501 | Samsung C27HG70 1440p 144hz HDR FreeSync 2 | Ubuntu 20.04.2 LTS |

 

Server:-

Intel NUC running Server 2019 + Synology DSM218+ with 2 x 4TB Toshiba NAS Ready HDDs (RAID0)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Sure, you can plan for the replacement but no amount of planning could have prevented the card from dying. And that was my thinking, if it was external then they coulda replaced it like for like, chances are they wouldn't have needed to even rebuild the array after the swap.

Come on people, watch the video! :)  He said that as it died, it messed up the array so swapping out for a new card probably wouldn't have helped.

Solve your own audio issues  |  First Steps with RPi 3  |  Humidity & Condensation  |  Sleep & Hibernation  |  Overclocking RAM  |  Making Backups  |  Displays  |  4K / 8K / 16K / etc.  |  Do I need 80+ Platinum?

If you can read this you're using the wrong theme.  You can change it at the bottom.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Kingston SSD's 

 

FrW84QF.jpg

 

 

 

 

also that hard drive pyramid in the video makes me nervous

Location: Kaunas, Lithuania, Europe, Earth, Solar System, Local Interstellar Cloud, Local Bubble, Gould Belt, Orion Arm, Milky Way, Milky Way subgroup, Local Group, Virgo Supercluster, Laniakea, Pisces–Cetus Supercluster Complex, Observable universe, Universe.

Spoiler

12700, B660M Mortar DDR4, 32GB 3200C16 Viper Steel, 2TB SN570, EVGA Supernova G6 850W, be quiet! 500FX, EVGA 3070Ti FTW3 Ultra.

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

In Big Enterprise environments we stopped using raid 3/4/5 a long time ago (Tons of reasons, start here). And even Raid 6 will be dead in the water:

http://queue.acm.org/detail.cfm?id=1670144

 

Card wise you had bad luck but storage should be done with JBOD Array & 2 SAS HBA's in Multipath Config and NO raid card. With Solaris and ZFS on top you'll get protection against URE, silent data corruption, bitflips etc. (Basically against all the RAID related crap). And you have protection against HW failures on every level (even if the whole storage server crashed).

 

Btrfs is not enterprise ready. 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

I thought the SSD-array server was running at least nightly backup to former freenas server?

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing LTT Releases

×