NANDpocalypse - 6 Exabytes lost

[The King of the Undead]

27 minutes ago, leadeater said:

It's actually worse if the ATS gets stuck in generator input because the arching welded it in place, you have no idea until it won't switch back to main then you're left having to leave the generator running for ages hoping it doesn't fail or run out of diesel before you can refill it, if you have one that is allowed to be refilled while running. We installed a large external fuel tank to get around that issue as much as possible.

 

ATS are losing favor because of that problem. Now I think common rail is getting more widely used and all power sources self balance using phase/frequency lead/lag so there is no power transfer switching at all.

your at the top for the last two pages and on the second to last your the top two.

[TechyBen]

16 hours ago, S w a t s o n said:

This sounds like complete bullshit to me. A 13 minute power outage has somehow ruined 6-9 Exabytes of nand wafers? I dont believe that for a second. This smells like a repeat of the single HDD factory flooding and everyone raising their prices because they could.

 

They said this is like half a quater's worth of production. You mean to tell me a 13 minute power outage ruined 1.5 months of wafers????

These systems can sometimes be on a month long processing. As in, any 1 second interruption to that process, and the wafer/chip fab (before being put on the PCB) is trashed. Entirely. Think of a photo development in the old school chemical developing. Any light shined on, and you trash the image. Same here, but for power outage on silicone chip fabrication.

 

The boxes/conveyor belts are all heated/cooled/protected and if left too long, the entire factory is "rinse and repeat" cleaning out the entire thing to start from scratch.

[Video Beagle]

15 hours ago, Lurick said:

it's far more expensive and time consuming to stop production for a period of time, no matter how short, and restart it all than it is to just keep churning things out and selling them for a loss

AH...this explains what Linus was saying on last week's lan show.  The calibration and validation etc...

[TechyBen]

52 minutes ago, leadeater said:

It's actually worse if the ATS gets stuck in generator input because the arching welded it in place, you have no idea until it won't switch back to main then you're left having to leave the generator running for ages hoping it doesn't fail or run out of diesel before you can refill it, if you have one that is allowed to be refilled while running. We installed a large external fuel tank to get around that issue as much as possible.

 

ATS are losing favor because of that problem. Now I think common rail is getting more widely used and all power sources self balance using phase/frequency lead/lag so there is no power transfer switching at all.

I think this happened here in the uk at some point. A 3 way backup central network link (with associated servers nearby) went down, partly because the doubled grid setup was ancient, and someone had plugged stuff in the wrong place (so not actually doubly protected) and the generators kicked in, but did not switch back to power when the power went back. XD

 

[Edit]

IIRC it was this one, or a similar incident. https://www.theregister.co.uk/2016/07/20/telecity_power_outage_bt_offline/

[paddy-stone]

17 hours ago, S w a t s o n said:

This sounds like complete bullshit to me. A 13 minute power outage has somehow ruined 6-9 Exabytes of nand wafers? I dont believe that for a second. This smells like a repeat of the single HDD factory flooding and everyone raising their prices because they could.

 

They said this is like half a quater's worth of production. You mean to tell me a 13 minute power outage ruined 1.5 months of wafers????

Yeah, apparently the factory only needs to run for 104 mins a year to supply their NAND needed   /s 

[leadeater]

3 minutes ago, paddy-stone said:

Yeah, apparently the factory only needs to run for 104 mins a year to supply their NAND needed   /s 

A job I only have to work for 104 minutes a year? Sign me up!

[Curious Pineapple]

Well it takes about 6 weeks from start to finish to produce silicon chips, 12 weeks in a quarter, you do the maths. It's a production line with all stages running at the same time, if a power outage affects the entire production line then everything from the crystals to the not-yet packaged devices are scrap.

 

If a power outage affected PCB assembly, anything that was at the pick and place stage would need to be started over, anything being reflowed may need to be scrapped as there is thermal profiles to adhere to.

 

Why the fuck does everything in the world need to be a conspiracy? No wonder shit for brains like Alex Jones get attention

[Lady Fitzgerald]

4 hours ago, leadeater said:

Yep, if you are building a 60MW capable power source then you're better off using it as primary and the grid as backup.

If you have the fuel reliably available at a reasonable cost.

 

I remember when the mine or railroad from the mine that provided coal to the large power plant in Page, AZ went on strike and the company I worked for had to contract gypsies to haul coal from the mine outside of Farmington, NM to keep the power plant up and running. I, along with other employees from where I worked, were sent up to monitor the coal haul (which was headed up by a crooked contractor that I exposed).

[TechyBen]

Cross post from The Register commentor:

Quote

oldtaku

Just 13 minutes

If you're wondering how they lose $600M of stuff in just 13 minutes, I do vacuum engineering work (as one of the hats).

Generally a setup like this is miles and miles of 'robots'. Not humanoid, but hexagonal with a chamber on each side. Each chamber exposes the wafer to things to build it up (gold), things to etch it, things to cure it. You roll up some wafers, the robot in the center moves one into chamber 1, does a process, then moves it from 1 to 2 and puts a new one in 1, etc, till all of the wafers have gone through the station and are ready for another combined process at the next station.

Critically a lot of these processes are done at low vacuum (like 10 mTorr) and often with toxic gases or worse, pyrophoric gasses that explode on contact with normal air, like silane. Everything is closely timed, and you have to carefully maintain 1) the pressure of the chamber, 2) the rate of incoming substance(s). If you cure the wafers for only 3 minutes instead of 5, you lost the wafer. Now into this happy little juggling act you throw a power loss.

*Honestly, it doesn't matter whether you lost if for 13 minutes or 13 seconds, you're done.*

Your CDGs that measure pressure generally take two hours to get back to correct internal temperature, so they're reading wrong. That doesn't really matter anyhow because your valves failed and you either put not enough gas into the chamber or way too much. If you put way too much in now your chamber is contaminated. And your vacuum pumps all failed, so you lost pressure control. The turbo pumps spin at 75000 RPM and can't handle any amount of thick gas, so maybe you bombed them (shattered the fans). The computers controlling these don't like being hard powered down.

Worse, and this is low probability and means you designed something wrong, but if you got too much silane and it contacted air because your pumps are down, maybe your robot caught on fire. Probably not, but either way you have to check all your turbos, open up all your robots, remove the destroyed wafers, clean your chambers. Oh, and now you need to recover all those process computers.

Nightmare scenario.

 

Add to that, some of these plants might take up 50% of a (small) normal power plants output, and there may be multiple industries in an area/city, then switching over to another grid could be a no-go too (spinning up the power need of multiple factories switching over is not possible if the power output of a power supply is not ready for it... it would take more than seconds! ).

[TheNamelessOne]

Can't have prices drop too much.

Need to find a way to raise them not to where they were, but beyond because we love MONEY and we hate Consumers, but we do love their money.

[Doobeedoo]

Oh wow the power downtime in mid of processing is severly bad. Hopefully this doesn't affect consumers much. 

[imreloadin]

Right on schedule...

[Delicieuxz]

6 hours ago, Lady Fitzgerald said:

Btw, supposition is not fact

Congratulations on recognizing the point, I guess.

 

Quote

and your own confirmation of supposition is worthless.

It's as worthless as you asking for somebody to back up a supposition - which was also the point, which I guess you didn't get after all.

 

Gonna have to take that congratulations back, then.

[TechyBen]

1 hour ago, Delicieuxz said:

Congratulations on recognizing the point, I guess.

 

It's as worthless as you asking for somebody to back up a supposition - which was also the point, which I guess you didn't get after all.

 

Gonna have to take that congratulations back, then.

No. No it's not. I can suppose you are out of your mind. Is that valuable?

 

[Lady Fitzgerald]

1 hour ago, Delicieuxz said:

Congratulations on recognizing the point, I guess.

 

It's as worthless as you asking for somebody to back up a supposition - which was also the point, which I guess you didn't get after all.

 

Gonna have to take that congratulations back, then.

[Delicieuxz]

3 minutes ago, TechyBen said:

No. No it's not. I can suppose you are out of your mind. Is that valuable?

Considering a supposition isn't somebody claiming something to be fact, why would there be a burden on them to back it up as if they were stating it as fact, when they weren't stating it as fact?

 

Gonna have to suppose you aren't very good with English. 

[TechyBen]

Just now, Delicieuxz said:

Considering a supposition isn't somebody claiming something to be fact, why would they back it up as if they were stating it as fact?

 

Gonna have to suppose you aren't very good with English. 

You placed that argument. Why make a supposition about facts? Should you not make suppositions about things not dependant on facts?

The other poster, was giving you an option to change your supposition to a backed opinion. You did not, instead doubling down on presumption.

[Delicieuxz]

14 minutes ago, TechyBen said:

You placed that argument. Why make a supposition about facts? Should you not make suppositions about things not dependant on facts?

The other poster, was giving you an option to change your supposition to a backed opinion. You did not, instead doubling down on presumption.

You and the other poster have misread things. 

 

What I playfully doubled-down on is that I presented a supposition as a supposition - and therefore the comment of my expression to be backed up is simply going to be that I had that impression. There isn't an argument within my comment that my impression is confirmed fact, and therefore such a claim (which I didn't make) isn't something for me to back-up.

 

14 minutes ago, TechyBen said:

You placed that argument.

It was a personal expectation presented with room for clarification. leadeater quickly clarified the matter, and I accepted their clarification.

 

Quote

Why make a supposition about facts? Should you not make suppositions about things not dependant on facts?

Every subject matter is ultimately factual. So, you're saying, 'why make suppositions?' The answer to that would be: To express personal thoughts, gain more information about things oneself isn't fully knowing of, raise questions, make opinionated points, etc.

 

I find it odd that my comment is stirring this much contention.

[Anomnomnomaly]

21 hours ago, WereCat said:

I call BS claims. 

 

If they lost 6Exabytes in 13min, they can make 6Exbytes in 13min which means that the production is so large that 6Exbytes doesn't matter since the production is so large that this is only a small fraction. 

In 13 minutes, they lost the current production run... that's stuff being created, stuff being prepared, processed and so forth.

 

So get the line back up and running, calibrated, test runs, quality control and so forth... can take a few weeks. This isn't a single line, this is dozens and dozens of them... and each one has to be restarted from scratch.

 

I have a friend who works making silicon wafer machines for the industry here in the UK and has been out to thailand, china and so forth to install and get them up and running.,, He confirmed it would take more than 2 weeks to start a line up.

[Lady Fitzgerald]

15 minutes ago, Delicieuxz said:

Considering a supposition isn't somebody claiming something to be fact, why would there be a burden on them to back it up as if they were stating it as fact, when they weren't stating it as fact?

 

Gonna have to suppose you aren't very good with English. 

 

13 minutes ago, TechyBen said:

You placed that argument. Why make a supposition about facts? Should you not make suppositions about things not dependant on facts?

The other poster, was giving you an option to change your supposition to a backed opinion. You did not, instead doubling down on presumption.

 

9 minutes ago, Delicieuxz said:

You and the other poster have misread things. 

 

What I playfully doubled-down on is that I presented a supposition as a supposition - and therefore the comment of my expression to be backed up is simply going to be that I had that impression. There isn't an argument within my comment that my impression is confirmed fact, and therefore such a claim (which I didn't make) isn't something for me to back-up.

 

It was a personal expectation presented with room for clarification. leadeater quickly clarified the matter, and I accepted their clarification.

 

Every subject matter is ultimately factual. So, you're saying, 'why make suppositions?' The answer to that would be: To express personal thoughts, raise questions, make opinionated points, etc.

 

I find it odd that my comment is stirring this much contention.

Either you are trolling or you do not know how to properly express yourself.

[Delicieuxz]

13 minutes ago, Lady Fitzgerald said:

Either you are trolling or you do not know how to properly express yourself.

That's called a false dichotomy. That actual cause of this fuss is that your reading comprehension has been less than adequate in this situation. You were simply mistaken and are offloading the responsibility for it onto me.

 

 

Does it help you to understand this sentence:

 

"For sure they have backup generators that are tested for reliability to make sure they work when needed."

 

... if the first words in it are modified like this?:

 

'Surely they have backup generators that are tested for reliability to make sure they work when needed.'

 

 

In either form, it is clear I am not claiming to be on the inside of their operations to know what they're doing as fact, but am venturing an assumption.

 

And, either way, the statement isn't worth this argument.

[S w a t s o n]

7 hours ago, TechyBen said:

These systems can sometimes be on a month long processing. As in, any 1 second interruption to that process, and the wafer/chip fab (before being put on the PCB) is trashed. Entirely. Think of a photo development in the old school chemical developing. Any light shined on, and you trash the image. Same here, but for power outage on silicone chip fabrication.

 

The boxes/conveyor belts are all heated/cooled/protected and if left too long, the entire factory is "rinse and repeat" cleaning out the entire thing to start from scratch.

Yea I get that wafers take time to be made but I dont think it should take 10 weeks (their claim) to process a wafer start to finish, especially NAND. I also doubt that the entirety of the wafers were being actively made and therefore fucked up. idk how many wafers it is exactly but 1.5 months of wafers is A LOT. I dont think anyone processes that many at once.

[JAKEBAB]

Lmao here we go again. Gotta keep those outages coming every few of months!

[Curious Pineapple]

51 minutes ago, S w a t s o n said:

Yea I get that wafers take time to be made but I dont think it should take 10 weeks (their claim) to process a wafer start to finish, especially NAND. I also doubt that the entirety of the wafers were being actively made and therefore fucked up. idk how many wafers it is exactly but 1.5 months of wafers is A LOT. I dont think anyone processes that many at once.

It takes 6 weeks to go from crystal to finished product in a hermetically sealed environment. You can't just half finish a wafer and store it in the shed out back, once started they have to be completed and as that takes about 6 weeks, they loose 6 weeks of product.

[Curious Pineapple]

2 hours ago, Anomnomnomaly said:

In 13 minutes, they lost the current production run... that's stuff being created, stuff being prepared, processed and so forth.

 

So get the line back up and running, calibrated, test runs, quality control and so forth... can take a few weeks. This isn't a single line, this is dozens and dozens of them... and each one has to be restarted from scratch.

 

I have a friend who works making silicon wafer machines for the industry here in the UK and has been out to thailand, china and so forth to install and get them up and running.,, He confirmed it would take more than 2 weeks to start a line up.

You could take half of these guys to a fab and show them how it works and they'll still scream that it's all a lie to increase the cost.