NANDpocalypse - 6 Exabytes lost

[straight_stewie]

On 6/28/2019 at 1:47 PM, iamdarkyoshi said:

Why wasn't this stuff battery backed?

Because fabs consume A LOT of electricity.

[Loote]

I don't think they will need to explain what made this fuckup possible, as was said before, it probably had a backup that failed and that's it. There's also a point where you'd wonder if it's possible to salvage some of the stuff that's supposedly garbage, I mean, 6 weeks production would mean every single piece was at a stage, at which power outage means destruction, it doesn't seem possible and maybe with $10m of investment they could continue working on $200m of chips instead of throwing them out. Whatever is the reason, I'd love to hear a good explanation when they find out what exactly happened and what is the cost of starting the work again.

To be honest, the way NAND prices look, it smells. The whole industry could probably gain more than was lost there if the news reports overstate the shortages etc. It would also be cool and scary it this is an effect of, for example, Samsung's spies' work.

[DeScruff]

On 6/28/2019 at 3:03 PM, iamdarkyoshi said:

True, but given how much this outage will cost them, you'd think they could afford a battery backup solution

With the amount of power required, to run a facility like this, I doubt "Batteries" would be feasible.
A Fab can use 60 Megawatts of power, backup generators wouldn't be enough, you essentially need a power plant, and I doubt anyone would think of building a backup power plant.

If I recall with a lot of semiconductors. it can take 8-12 weeks to manufacture each wafer. So its not like they manufacture 6 exabytes per 12 min, Its that the NAND they have been manufacturing for the past 8-12 weeks is 6 exabytes worth is damaged because air purifiers are stopped, machines that turned of unexpectedly in the middle of a job may need to be re-calibrated exc.

It be like if a farmer put all their produce in a giant fridge, had a power out, and left at summer temperatures for a week. - They might have just lost a whole months worth of produce in a much shorter amount of time, and it will take time to recover all that.

[Lady Fitzgerald]

6 hours ago, Delicieuxz said:

That's called a false dichotomy. That actual cause of this fuss is that your reading comprehension has been less than adequate in this situation. You were simply mistaken and are offloading the responsibility for it onto me.

 

 

Does it help you to understand this sentence:

 

"For sure they have backup generators that are tested for reliability to make sure they work when needed."

 

... if the first words in it are modified like this?:

 

'Surely they have backup generators that are tested for reliability to make sure they work when needed.'

 

 

In either form, it is clear I am not claiming to be on the inside of their operations to know what they're doing as fact, but am venturing an assumption.

 

And, either way, the statement isn't worth this argument.

[Unixsystem]

3 hours ago, Loote said:

I don't think they will need to explain what made this fuckup possible, as was said before, it probably had a backup that failed and that's it. There's also a point where you'd wonder if it's possible to salvage some of the stuff that's supposedly garbage, I mean, 6 weeks production would mean every single piece was at a stage, at which power outage means destruction, it doesn't seem possible and maybe with $10m of investment they could continue working on $200m of chips instead of throwing them out. Whatever is the reason, I'd love to hear a good explanation when they find out what exactly happened and what is the cost of starting the work again.

If I had to guess, the amount "lost" is probably not wafers that were actually destroyed, but the loss of production capacity that results from needing to keep the factory offline for a few weeks and validate that everything is still configured. I'm sure plenty of wafers were actually literally lost due to being partway through an intricate process that can't be easily restarted, but I'd imagine the downtime is much more impactful than the material cost. 

[Delicieuxz]

1 hour ago, Lady Fitzgerald said:

My thoughts exactly.

[Suika]

3 hours ago, Waffles13 said:

If I had to guess, the amount "lost" is probably not wafers that were actually destroyed,

No, wafers would have to be a good part of the cost. Samsung previously lost 60,000 wafers from a single fab outtage, but here we're talking about five fabs and an R&D center. That'd be a lot of potential product ruined because of an interrupted manufacturing process.

[jagdtigger]

First they arbitrarily puffed up HDD prices and now this happens just as SSD's gett;ng pretty cheap.....

[realpetertdm]

Toshiba and WD: gets rekt by a power outage

Samsung: 

 

 

[TechyBen]

14 hours ago, Delicieuxz said:

That's called a false dichotomy. That actual cause of this fuss is that your reading comprehension has been less than adequate in this situation. You were simply mistaken and are offloading the responsibility for it onto me.

 

 

Does it help you to understand this sentence:

 

"For sure they have backup generators that are tested for reliability to make sure they work when needed."

 

... if the first words in it are modified like this?:

 

'Surely they have backup generators that are tested for reliability to make sure they work when needed.'

 

 

In either form, it is clear I am not claiming to be on the inside of their operations to know what they're doing as fact, but am venturing an assumption.

 

And, either way, the statement isn't worth this argument.

And the question was "can you back that up." or as you put it, what if we '... if the first words in it are modified like this?:'

 

Quote

Can you learn what is actually in the factory?

You made a supposition. A really really big one. Like, without any data or knowledge. I think the other poster was questioning such a dangerous method of thought.

[TechyBen]

[deleted due to browser bug double post.]

[CarlBar]

@leadeater Ahh should have explained the UK national grid thing, sorry. I was thinking specifically of a [articular power plant we have in i think it's either Scotland or Wales, can't remember which without digging into it again. Not only is it a really high output, (multi-gigawatt), peaking plant with a really fast spin up time, (at worst 60 seconds from being told to go if completely cold and no warning, 15 seconds if given a tiny bit of warning), but it's one of the primary plant options for restarting the grid from a black start condition, that is if the grid crashes out and the rest of the power plants that do normal baseline loading are forced to shut down it's a primary choice for getting them power so they can restart their plant machinery and synchronise the generator sets to restart their power production. It's effectively also a backup generator for the entire grid.

 

 

[TechyBen]

45 minutes ago, CarlBar said:

@leadeater Ahh should have explained the UK national grid thing, sorry. I was thinking specifically of a [articular power plant we have in i think it's either Scotland or Wales, can't remember which without digging into it again. Not only is it a really high output, (multi-gigawatt), peaking plant with a really fast spin up time, (at worst 60 seconds from being told to go if completely cold and no warning, 15 seconds if given a tiny bit of warning), but it's one of the primary plant options for restarting the grid from a black start condition, that is if the grid crashes out and the rest of the power plants that do normal baseline loading are forced to shut down it's a primary choice for getting them power so they can restart their plant machinery and synchronise the generator sets to restart their power production. It's effectively also a backup generator for the entire grid.

 

 

Not sure if it's the same thing or different, but due to the hills/landscape they do have at least 1 power "store" in the form of a pumping/generator lake/dam. They can offset overproduction and pump up water, then let it go back down through the generators to regain it back later. That kind of setup could be ramped up quickly (opening a flow valve). But the places factories are, might be flat.

[leadeater]

2 hours ago, TechyBen said:

That kind of setup could be ramped up quickly (opening a flow valve). But the places factories are, might be flat.

Really really big and tall water towers 

[Loote]

14 hours ago, Waffles13 said:

If I had to guess, the amount "lost" is probably not wafers that were actually destroyed, but the loss of production capacity that results from needing to keep the factory offline for a few weeks and validate that everything is still configured. I'm sure plenty of wafers were actually literally lost due to being partway through an intricate process that can't be easily restarted, but I'd imagine the downtime is much more impactful than the material cost. 

Oh, I was thinking of it in terms of material that they paid for lost, not 'lost income' that anti piracy guys love, but from that point of view it seems much more feasible.

 

 

https://www.theguardian.com/technology/2018/sep/27/south-australias-tesla-battery-on-track-to-make-back-a-third-of-cost-in-a-year
 

I wonder if there won't be a wave of upgrades with that sort of batteries in all the fabs, 127MWh for 90M Australian dollars, or $31M for 60 MW battery that would run for an hour,

[CarlBar]

5 hours ago, TechyBen said:

Not sure if it's the same thing or different, but due to the hills/landscape they do have at least 1 power "store" in the form of a pumping/generator lake/dam. They can offset overproduction and pump up water, then let it go back down through the generators to regain it back later. That kind of setup could be ramped up quickly (opening a flow valve). But the places factories are, might be flat.

 

It is of that type but we've got quite a few like it. it's just the biggest of them.

[Delicieuxz]

13 hours ago, TechyBen said:

And the question was "can you back that up." or as you put it, what if we '... if the first words in it are modified like this?:'

I used "for sure" as expression like "surely", the definition for which includes:

 

 

The question to "back that up" mistook my comment as a statement of verified fact rather than my personal impression. Though, I hadn't yet read Lady Fitzgerald's previous comments giving insight into the matter.

 

I pointed out that it is not for me to back up something I didn't claim (such as it's fact that the company would have backup power) in my next post, which further clarified my statement as my own impression:

On 6/28/2019 at 8:10 PM, Delicieuxz said:

It's a supposition, so backing it up would simply be me confirming that I assert it. I confirm it.

 

Much smaller companies have backup power to protect their operations. I think it would be pretty crazy if a large tech production process was running at the mercy of the unknown. Kind of a big oversight for an advanced tech company.

I think that there shouldn't have been any further confusion over things after that point.

 

Quote

You made a supposition. A really really big one. Like, without any data or knowledge.

And that's why I don't know what any contention is over, because that's exactly what a supposition is:

 

 

 

 

Quote

I think the other poster was questioning such a dangerous method of thought.

I don't think that normally expressing oneself in conversation becomes dangerous. But then, I don't know how you are seeing what I said. It's just a misunderstanding.

[TechyBen]

14 hours ago, Delicieuxz said:

I used "for sure" as expression like "surely", the definition for which includes:

Spoiler

 

 

The question to "back that up" mistook my comment as a statement of verified fact rather than my personal impression. Though, I hadn't yet read Lady Fitzgerald's previous comments giving insight into the matter.

 

I pointed out that it is not for me to back up something I didn't claim (such as it's fact that the company would have backup power) in my next post, which further clarified my statement as my own impression:

I think that there shouldn't have been any further confusion over things after that point.

 

And that's why I don't know what any contention is over, because that's exactly what a supposition is:

 

 

 

 

I don't think that normally expressing oneself in conversation becomes dangerous. But then, I don't know how you are seeing what I said. It's just a misunderstanding.

 

 

Surely!

Surely?

 

I know what you did. You're supposedly doubling down!

I know what you did. You're supposedly doubling down?

 

No worries. We mistook it because of a misunderstanding. But you are doubling down it was our error. We don't mind either way. But if only we make note of it, you may repeat that error.

[Delicieuxz]

46 minutes ago, TechyBen said:

Surely!

Surely?

 

I know what you did. You're supposedly doubling down!

I know what you did. You're supposedly doubling down?

 

No worries. We mistook it because of a misunderstanding. But you are doubling down it was our error. We don't mind either way. But if only we make note of it, you may repeat that error.

If you look, the matter of an exclamation vs question mark does not change the applicability of 'surely' as an expression, as illustrated in the above dictionary example of "Surely the problem lies in the design of the equipment", and also in this plethora of examples:

 

[TechyBen]

Yes. I agree. But that does not clear up the discussion. Talking about what we mean does.

Doubling down on it, instead of saying "I made an assumption based on no information", kind of is strange.

Confirming you made an assumption based on no information, and that it correctly was an assumption based on no information, is a strange response when people ask "why not back that up instead of making an assumption based on no information?"

 

Yes, you were not clear on exactly if you were asking a question or making a statement (it's a double meaning word!), and we were not entirely clear on if we were asking you to back up a supposition, or to go and stop making suppositions, and learn something about the situation.

 

So, after all that, are we to continue to defend the right to make suppositions, or understand that progressing from a supposition to a bit of extra knowledge is better? A supposition is great, only if used to then find out where to start learning.

 

If we just post a supposition, on its own. Of what use or purpose was the post?

 

[edit]

PS, and the answer to all of those examples you gave, might be "surely you could back that up with facts?" and no one would argue against it. Those are all questions/statements, and all of them could require or be responded by someone asking "please provide proof/why do you think that?".

[CarlBar]

3 hours ago, TechyBen said:

Yes. I agree. But that does not clear up the discussion. Talking about what we mean does.

Doubling down on it, instead of saying "I made an assumption based on no information", kind of is strange.

Confirming you made an assumption based on no information, and that it correctly was an assumption based on no information, is a strange response when people ask "why not back that up instead of making an assumption based on no information?"

 

Yes, you were not clear on exactly if you were asking a question or making a statement (it's a double meaning word!), and we were not entirely clear on if we were asking you to back up a supposition, or to go and stop making suppositions, and learn something about the situation.

 

So, after all that, are we to continue to defend the right to make suppositions, or understand that progressing from a supposition to a bit of extra knowledge is better? A supposition is great, only if used to then find out where to start learning.

 

If we just post a supposition, on its own. Of what use or purpose was the post?

 

[edit]

PS, and the answer to all of those examples you gave, might be "surely you could back that up with facts?" and no one would argue against it. Those are all questions/statements, and all of them could require or be responded by someone asking "please provide proof/why do you think that?".

 

Whilst i agree with the point regarding his recent responses i can kinda understand why he'd make the supposition he did. There;s a lot of places, (i mentioned a couple, leadenter got one and i'm sure there are a few more), where having backup power really really makes sense even when it's a high power application because the consequences of letting the power dropout without a backup are so severe. And it's honestly surprising that this isn't one of them at first glance.Until i saw people doing some serious digging into the numbers i had my doubts about the claims of no backup generators TBH.

[Syntaxvgm]

On 6/28/2019 at 2:47 PM, iamdarkyoshi said:

Why wasn't this stuff battery backed?

If it gives you an idea, I just sold a couple of new switches earlier this year with fuses the size of sausages, and they'll only run one piece of equipment. Rated to a 60hp motor I think. 

You can run production off of backup power unless you have lines from 2 utility providers, even then it's often not practical to set that up. 

[TechyBen]

18 hours ago, CarlBar said:

 

Whilst i agree with the point regarding his recent responses i can kinda understand why he'd make the supposition he did. There;s a lot of places, (i mentioned a couple, leadenter got one and i'm sure there are a few more), where having backup power really really makes sense even when it's a high power application because the consequences of letting the power dropout without a backup are so severe. And it's honestly surprising that this isn't one of them at first glance.Until i saw people doing some serious digging into the numbers i had my doubts about the claims of no backup generators TBH.

Yep. Sorry about that. Many many HDD factories have similar problems. Many other types of factories can. Some facilities, like car manufacturing don't (I would guess the paint sections may cause problems for those panels).

 

Some factories can stop on a dime and restart. Others cannot (aluminum/metal work give as an example above). And some of those have dedicated power stations. So if the power station goes down... well, do we expect every factory to build twice the power stations they need? XD

 

I'd rather lose a friend, than risk watching someone defend their choice not to learn. I hope Delicieuxz would look into a situation, when someone asks "can you back that up?" and not take such a request as an offense, when it was a statement asking them to learn.

[79wjd]

On 6/29/2019 at 7:23 AM, Dacien said:

That's fascinating. I guess it is an unstoppable process. They sell generators with megawatt-level output, but I also vastly underestimated the power demands of these plants.

 

That's fascinating to me that they just accept power outages and major loss as a cost of doing business. That's amazing to me.

 

 

One word:

 

Insurance

[TechyBen]

But it's the same in many industries. You cannot stop the rain, only mitigate it. So farming is based around that. The costs of putting every plant in a greenhouse are prohibitive, even if it guarantees "environment".

 

Similar here. Someone did a cost/return check, it was not in it for them (though they can still make errors there, and I doubt they were 100% correct here). TBF, it's their factory. If they wish to take the short term loss, vs long term costs (of a power plant), or wish to take the "risks", then it is their choice.