NANDpocalypse - 6 Exabytes lost

[leadeater]

Quote

Emergency and Backup Power

 

A supply of fluctuation-free electricity is critical. Chip fabrication plants and server farms must balance the expense of building independent electricity resources with the cost of equipment failures and network crashes caused by unreliable power. Hewlett-Packard has estimated that a 15-minute outage at a chip fabrication plant cost the company $30 million, about half the plant’s power budget for a year.   Backup systems are so expensive, that a survey of 48 companies revealed only 3 had backup power sources: 3 used generators and the other one solar (Hordeski).

 

It’s too expensive to operate a separate power plant to generate power. Fab plants use up to 60 megawatts of power, so putting a natural gas or coal power plant onsite would cost somewhere between $100-400 million dollars.

http://energyskeptic.com/2014/interdependent-chip-fab-electricgrid-financial-sys/

 

Perspective on how many actually have backup power.

[Beskamir]

Welp, there go the decreasing prices on nand storage.

[RejZoR]

It was looking too good for SSD prices so they had to inflate them by some sort of incident. If it ain't flood it's power outage coz hamsters in power department were on stike or some shit.

[mr moose]

11 hours ago, TetraSky said:

How do you somehow lose 6EB of NAND in 13 minutes without power?

 

When the lights came back on the only thing they saw was a man in striped cloths and a black mask holding a black bag over his right should sneaking out of the door on his tippy toes.

 

 

 

 

 

[kaiju_wars]

Might be the time to stock up on some SSDs.  I always like to have extras, and I need some for some projects anyways.  

Damn, just as they were falling in price and I was thinking of phasing out my HDDs for some 1TB or 2TB SSDs.

[RejZoR]

21 minutes ago, mr moose said:

When the lights came back on the only thing they saw was a man in striped cloths and a black mask holding a black bag over his right should sneaking out of the door on his tippy toes.

 

 

 

 

 

God damn raccoons and their shenanigans.

[Tellos]

Lets remember they also run a clean room likely for sedative parts. This means they have to restart and ensure the airs dust free and continent free during this time. I;ve gotten to go inside once to a small fabs clean room long ago for production. They are VERY particular about what gets touched by what to ensure no damage to sensitive parts. Likely now they have to check any salvageable pieces or parts for continents that may now have ruined material.

[leadeater]

4 minutes ago, Tellos said:

Lets remember they also run a clean room likely for sedative parts. This means they have to restart and ensure the airs dust free and continent free during this time. I;ve gotten to go inside once to a small fabs clean room long ago for production. They are VERY particular about what gets touched by what to ensure no damage to sensitive parts. Likely now they have to check any salvageable pieces or parts for continents that may now have ruined material.

I'd also be worried if they find large areas of earth the size of Australia inside the fab facility, why do Aussies have to destroy everything.

 

Sorry couldn't resist.

[Tellos]

4 minutes ago, leadeater said:

I'd also be worried if they find large areas of earth the size of Australia inside the fab facility, why do Aussies have to destroy everything.

 

Sorry couldn't resist.

I dont get the reference or joke.

[leadeater]

4 minutes ago, Tellos said:

I dont get the reference or joke.

Continent vs Contaminant

[Tellos]

4 minutes ago, leadeater said:

Continent vs Contaminant

Sorry spell check apparently got it wrong.

[justpoet]

4 hours ago, leadeater said:

http://energyskeptic.com/2014/interdependent-chip-fab-electricgrid-financial-sys/

 

Perspective on how many actually have backup power.

I would argue that more probably do have some form of UPS/battery backup power in a few production locations, but only enough to let a production machine halt without damaging itself, rather than trying to continue production or save product.

[leadeater]

1 minute ago, justpoet said:

I would argue that more probably do have some form of UPS/battery backup power in a few production locations, but only enough to let a production machine halt without damaging itself, rather than trying to continue production or save product.

Yea that stat was just looking at generators I believe, most have UPSs even just for the power quality protection they offer.

[RejZoR]

It's weird that company like Toshiba doesn't have worked out power redundancy. If power outagge results in such massive production defect, I'd be damn sure to have this ready at all times.

[Lady Fitzgerald]

6 hours ago, Delicieuxz said:

It's a supposition, so backing it up would simply be me confirming that I assert it. I confirm it.

 

Much smaller companies have backup power to protect their operations. I think it would be pretty crazy if a large tech production process was running at the mercy of the unknown. Kind of a big oversight for an advanced tech company.

You haven't been paying attention. Backup power generation doesn't scale up well, both in cost effectiveness and biltiy to get online. The amount of generation required would cost more than the losses incurred from an outage. Also, the larger the generation plant required, the longer it takes to bring it online. Reread what I already wrote on that.

 

Btw, supposition is not fact and your own confirmation of supposition is worthless.

5 hours ago, leadeater said:

http://energyskeptic.com/2014/interdependent-chip-fab-electricgrid-financial-sys/

 

Perspective on how many actually have backup power.

Excellent article that did a more detailed account of what I wrote earlier. One thing not mentioned was the amount of time it takes to get generation units of the size needed online from a cold start. The only way for plants large enough to power a massive semiconductor manufacturing plant would be to keep the plant idling. Even then, it takes a while to ramp up production from even idle, even though it is faster than a cold start up. Eating the cost of an outage is going to be far less than the cost of building a large enough power plant, maintaining it, and keeping it idling at all times.

2 hours ago, kaiju_wars said:

Might be the time to stock up on some SSDs.  I always like to have extras, and I need some for some projects anyways...

I do the same thing. I keep at least one spare of every size and type of SSD is have (which is quite a few since I abandoned HDDs a few years ago) since, despite living in a megalopolis (Phoenix, AZ area), replacements are not available locally and the online vendors I trust to buy SSDs from occasionally run out of stock.

19 minutes ago, RejZoR said:

It's weird that company like Toshiba doesn't have worked out power redundancy. If power outagge results in such massive production defect, I'd be damn sure to have this ready at all times.

Go back over this thread again.

[Faisal A]

Good job I just bought my SSDs a few days ago

[NunoLava1998]

Now, Optane SSDs will be a little less ridiculous compared to NVMe drives.

Intel executives are likely partying right now, considering their Optane SSDs will make their first sale in years.

[leadeater]

7 minutes ago, Lady Fitzgerald said:

Excellent article that did a more detailed account of what I wrote earlier. One thing not mentioned was the amount of time it takes to get generation units of the size needed online from a cold start. The only way for plants large enough to power a massive semiconductor manufacturing plant would be to keep the plant idling. Even then, it takes a while to ramp up production from even idle, even though it is faster than a cold start up. Eating the cost of an outage is going to be far less than the cost of building a large enough power plant, maintaining it, and keeping it idling at all times.

Yep, if you are building a 60MW capable power source then you're better off using it as primary and the grid as backup.

[RejZoR]

Yeah, it's especially lower costif you can just dump the costs of failure on the consumers who just have to accept it. Like they always do.

[WkdPaul]

8 hours ago, Lady Fitzgerald said:

Can you back that up?

I work for a small IT company, with less than 60 employees and on top of the battery backup, we also have a generator that is inspected every day ... now, if a manufacturing plant of that size doesn't have generators, then someone somewhere hasn't done their job properly.

 

With that said, it's 100% possible that the generators didn't kick in for some reason, that happened to us once, despite the daily inspections, incidents do happen.

 

 

EDIT :

 

just saw the article @leadeater posted, yeah, with that amount of power, I can understand why the whole plant isn't backed up. But I feel like there should be a middle ground, no? (at least, it looks like they had a some battery backup power, I'm just surprised there's no other form of backup, even only on specific critical applications). 

Edited June 29, 2019 by wkdpaul

[Arika]

14 minutes ago, wkdpaul said:

I work for a small IT company, with less than 60 employees and on top of the battery backup, we also have a generator that is inspected every day ... now, if a manufacturing plant of that size doesn't have generators, then someone somewhere hasn't done their job properly.

 

With that said, it's 100% possible that the generators didn't kick in for some reason, that happened to us once, despite the daily inspections, incidents do happen.

I work in a large office building that has 2 massive diesel generators that will kick in to keep system critical things online. since they take about 30ish seconds to hit full output, we have battery backups to run things until the generator takes over, technically we can run for 48 hours without power, but that's literally only powering computers and basic servers. our data centre is offsite. that would be an insane amount of power and generators to keep a foundry running for even a few minutes.

[Dacien]

7 hours ago, leadeater said:

http://energyskeptic.com/2014/interdependent-chip-fab-electricgrid-financial-sys/

 

Perspective on how many actually have backup power.

That's fascinating. I guess it is an unstoppable process. They sell generators with megawatt-level output, but I also vastly underestimated the power demands of these plants.

 

That's fascinating to me that they just accept power outages and major loss as a cost of doing business. That's amazing to me.

 

 

[leadeater]

2 minutes ago, Dacien said:

I also vastly underestimated the power demands of these plants.

So did I until I looked in to it, was thinking like 5MW max, way off lol.

[leadeater]

51 minutes ago, wkdpaul said:

With that said, it's 100% possible that the generators didn't kick in for some reason, that happened to us once, despite the daily inspections, incidents do happen.

It's actually worse if the ATS gets stuck in generator input because the arching welded it in place, you have no idea until it won't switch back to main then you're left having to leave the generator running for ages hoping it doesn't fail or run out of diesel before you can refill it, if you have one that is allowed to be refilled while running. We installed a large external fuel tank to get around that issue as much as possible.

 

ATS are losing favor because of that problem. Now I think common rail is getting more widely used and all power sources self balance using phase/frequency lead/lag so there is no power transfer switching at all.

[orbitalbuzzsaw]

Good thing I stocked up on SSDs already.