The chip shortage strikes again - Microsoft runing out of computers in its Azure UK datacentres

[colonel_mortis]

Summary

Following a spike in demand, Microsoft is no longer accepting new customers for virtual machines or Cosmos DB in either of its Azure UK regions. They are blaming the chip shortage for limiting their ability to scale up to meet an unexpected spike in demand.

 

Quotes

Quote

Microsoft’s flagship cloud service has stopped accepting new customers who want a UK-based service as the business is squeezed by emergency demand [...]. A global microchip shortage means it cannot easily expand capacity to take on more clients.

 

Microsoft’s flagship Azure cloud computing service has two data centre regions in Britain, called UK South and UK West. Neither is accepting new customers for two of its most widely-used services, Cosmos DB and virtual machines.

 

A message from Microsoft to QuoStar, seen by the Telegraph, said: “Unfortunately, due to high demand in this region (UK South), we are not able to approve your request at this time". 

It went on to offer a “bi-monthly” update on whether services would be available in future.

 

My thoughts

I thought the chip shortage was supposed to be mostly resolved at this point, and I would have imagined that a company with the purchasing power of Microsoft would be one of the last to be affected. I guess this has happened because they have gradually lost headroom over the past couple of years as they haven't been able to scale quite as much as they wanted, coupled with an unexpected spike in demand recently. I'm surprised that they are talking about months before they will be able to catch up though.

 

The big tech companies have done an impressive job of keeping ahead of demand until now, and have maintained the illusion that they can scale as far as they like. It looks like Microsoft has decided to protect that illusion for their existing customers, at the expense of new customers, until they can catch up with capacity.

 

Sources

https://www.telegraph.co.uk/business/2022/07/02/microsoft-declines-new-cloud-customers-promise-ukraine/

[porina]

Shortage is easing but I don't think it is over. Part of the problem may be product mix too. The various ongoing world situations have made demand more difficult to predict. Semiconductors are relatively long lead time even if you have capacity, so you can't just react quickly if forecasts are off.

[leadeater]

57 minutes ago, colonel_mortis said:

I thought the chip shortage was supposed to be mostly resolved at this point, and I would have imagined that a company with the purchasing power of Microsoft would be one of the last to be affected.

Sadly no and it's not actually CPUs and GPUs that are the problem. We have delays on HPE servers of over 180 days if you have a configuration that needs an add-in HBA/RAID card, as most HBA/RAID cards are manufactured in Ukraine, go figure I never knew until now. Some models aren't and manufacturing is being started in other places.

 

Anyway, other parts are affected too like drive bay backplanes. Unclear if same reason as above. Server PSU's over 800W are also affected.

 

57 minutes ago, colonel_mortis said:

I'm surprised that they are talking about months before they will be able to catch up though.

New parts suppliers and models are in the process of being certified right now, at least that's what HPE are doing and I'd bet everyone else is too. HPE ProLiant is expected to have this completed soon, followed by the next most popular product lines.

[Rauten]

Reminds me of a community post by Square-Enix in FF14, about how they wanted to increase their server capacity but couldn't, because there just weren't any servers they could buy!

They have since been able to expand somewhat, but not as much as they wanted.

 

The car industry is also still suffering shortages. The other day I saw a news article in my company's intranet about a certain company having a large amount of half-finished products due to lack of materials to finish them, mostly tech-related.

[LAwLz]

A few months ago I had someone tell me, with a straight face, that the shortages were caused by crypto miners and that no other industry except gaming GPUs were affected...

 

Anyone who is slightly involved with the industry (and no, buying graphics cards to play games is not being involved) could have told you that we were heading this way months ago. Lead times on new severers have not exactly going down. 

Want a new sever? You better order it today if you want it next year. Even servers without GPUs. 

 

Shipping issues. 

Lockdowns. 

Material shortages. 

Restrictions on imports/exports of things like noble gas from Russia. 

High inflation. 

High demand. 

 

All factors of why things are the way they are. 

 

 

At this point I just hope prices will return to somewhat normal during 2023.

[Sauron]

17 hours ago, colonel_mortis said:

I thought the chip shortage was supposed to be mostly resolved at this point

Absolutely not, in fact it's getting worse. New chips are less of an issue but anything that requires legacy chips is SOL. The company I work for is currently scouring ebay for the hardware we need in machines and paying upwards of 20 times the regular price for used units. We'll probably manage this year but if this doesn't let up I don't know what will happen next year.

[suicidalfranco]

On 7/6/2022 at 4:50 PM, LAwLz said:

A few months ago I had someone tell me, with a straight face, that the shortages were caused by crypto miners and that no other industry except gaming GPUs were affected...

 

Anyone who is slightly involved with the industry (and no, buying graphics cards to play games is not being involved) could have told you that we were heading this way months ago. Lead times on new severers have not exactly going down. 

Want a new sever? You better order it today if you want it next year. Even servers without GPUs. 

 

Shipping issues. 

Lockdowns. 

Material shortages. 

Restrictions on imports/exports of things like noble gas from Russia. 

High inflation. 

High demand. 

 

All factors of why things are the way they are. 

 

 

At this point I just hope prices will return to somewhat normal during 2023.

Nah m8, it's the miners fault I can't play cod 69

[MageTank]

On 7/5/2022 at 6:53 PM, leadeater said:

Sadly no and it's not actually CPUs and GPUs that are the problem. We have delays on HPE servers of over 180 days if you have a configuration that needs an add-in HBA/RAID card, as most HBA/RAID cards are manufactured in Ukraine, go figure I never knew until now. Some models aren't and manufacturing is being started in other places.

 

Anyway, other parts are affected too like drive bay backplanes. Unclear if same reason as above. Server PSU's over 800W are also affected.

 

New parts suppliers and models are in the process of being certified right now, at least that's what HPE are doing and I'd bet everyone else is too. HPE ProLiant is expected to have this completed soon, followed by the next most popular product lines.

 

On 7/6/2022 at 10:50 AM, LAwLz said:

A few months ago I had someone tell me, with a straight face, that the shortages were caused by crypto miners and that no other industry except gaming GPUs were affected...

 

Anyone who is slightly involved with the industry (and no, buying graphics cards to play games is not being involved) could have told you that we were heading this way months ago. Lead times on new severers have not exactly going down. 

Want a new sever? You better order it today if you want it next year. Even servers without GPUs. 

 

Shipping issues. 

Lockdowns. 

Material shortages. 

Restrictions on imports/exports of things like noble gas from Russia. 

High inflation. 

High demand. 

 

All factors of why things are the way they are. 

 

 

At this point I just hope prices will return to somewhat normal during 2023.

Us in the system integration business are actually facing massive shortages in wireless adapters. We order OEM boards from every brand under the sun and they can arrive with different wireless adapters. This means different FCCID labels are required per board configuration and depending on whether it ships with an inbox driver, a potential change in system image. It's also annoying when you order a few thousand boards at a time and have to sort through them to determine which board uses which system image because board vendors decided not to rev their boards with down-level component change. At the very least, we haven't encountered downgrading their WiFi 6 boards to WiFi 5 without telling people... yet.

 

I never would have expected a WiFi adapter shortage to be my crux after dealing with the GPU shortage, lol.

[LAwLz]

18 hours ago, MageTank said:

 

Us in the system integration business are actually facing massive shortages in wireless adapters. We order OEM boards from every brand under the sun and they can arrive with different wireless adapters. This means different FCCID labels are required per board configuration and depending on whether it ships with an inbox driver, a potential change in system image. It's also annoying when you order a few thousand boards at a time and have to sort through them to determine which board uses which system image because board vendors decided not to rev their boards with down-level component change. At the very least, we haven't encountered downgrading their WiFi 6 boards to WiFi 5 without telling people... yet.

 

I never would have expected a WiFi adapter shortage to be my crux after dealing with the GPU shortage, lol.

One of my clients makes controller boards for fairly niche industrial equipment. 

Because of the drastic shortages for things like certain IC chips some staff members have been reassigned from assembling, to just looking at sites like eBay after components.

Then if they find some component they need they order it from whichever seller they can, usually at like 5 to 10 times higher than the usual price. 

 

It gives pretty good insight into what a massive domino effect certain things have. 

A factory in China shut down because of covid.

This lead to a certain IC being out of stock.

Now a company can't build a controller board because of that missing IC.

That in turn means that a different company can't build their forklift which uses they controller board. 

That means that a third company are now several forklifts short and are not operating at 100% efficiency. 

 

And the spiral goes on... 

And that's just one company. The same is happening to thousands of companies right now, with tens of thousands of components being in short supply and high demand. 

[IkeaGnome]

13 minutes ago, LAwLz said:

And that's just one company. The same is happening to thousands of companies right now, with tens of thousands of components being in short supply and high demand.

We have an electric motor. Fairly common one for this brand to use on their mining drills. 3 years ago, they'd have one stock in town and we'd have it next week on the barge. We just got great news on it yesterday. None in stock anywhere. Stockholm(the companies HQ) say we'll be lucky to have it by December with current lead times on their suppliers.

[Laborant]

Working with FPGAs is also some sort of Nightmare. Some weeks ago, the "Embedded World Conference" in Nuremberg (Germany) took place, and we as a research team of our University went there. There we went to an FPGA manufacturer and asked them, if they have reasonable Developer Kits, we could use in our Bachelor's courses for student laboratory exercises.

Quote: "I give you this contact, if you need 50 Developer Kits, he is your guy. But don't expect short lead times. Currently we ourselves do not have 50 of our own chips. Lately I gave away a privately owned DevKit to two begging students."

The same with Nvidia Jetson Modules of all sort. (Nvidia Jetson are the "ARM-SOC + associated GPU" lineup for embedded computing, somehow targeting Automotive and Industrial applications. Think of it as a Nintendo Switch with current-gen hardware where you have to attach your own controls and screen.)

To me, it seems like the entire high-tech economy being optimized like an Olympic sprinter, and somehow COVID threw a broomstick in between his legs. 

[Mihle]

In industry there is problems getting some electrical components on components that has usually always been in stock. For example some frequency converters you now get a few months after you order them as an example. I am guessing it is also because of chips shortage.

[Kisai]

On 7/6/2022 at 4:07 AM, Rauten said:

Reminds me of a community post by Square-Enix in FF14, about how they wanted to increase their server capacity but couldn't, because there just weren't any servers they could buy!

They have since been able to expand somewhat, but not as much as they wanted.

 

The car industry is also still suffering shortages. The other day I saw a news article in my company's intranet about a certain company having a large amount of half-finished products due to lack of materials to finish them, mostly tech-related.

This problem predates everything.

 

Before Covid even hit, there was already a 3 month or more lead time to get hardware orders from HP and Dell, and policies regarding how long hardware was supposed to be maintained got stretched from replacement every 3 years to "5 if in good condition" kind of things. Sadly, most laptops rarely survive 1 year under the best conditions, usually the cooling fans need to be replaced after 9 months because the users use the laptops for 8 hours a day 5 days a week, and the laptops are not turned off when they go home. I'm not kidding that "ultrabooks" should have no moving parts, because that would ensure their lifespan would be that of the SSD wear or monitor brightness instead.

 

On the server end of things, two of my clients still have PowerEdge 1950's in service. It's really a "if it ain't broke, don't fix it" situation with those. Other hardware that a client was using using, we waited for months for cisco to ship something.

[shadow_ray]

5 minutes ago, Kisai said:

Sadly, most laptops rarely survive 1 year under the best conditions, usually the cooling fans need to be replaced after 9 months because the users use the laptops for 8 hours a day 5 days a week, and the laptops are not turned off when they go home.

What do you mean by "survive"? If it only suffers from a minor issue do you count it as dead? One year is a really short period of time.

 

Why they keep the machines running? I really don't understand. Mine goes to sleep after 30 mins or so by default.

[Rauten]

24 minutes ago, Kisai said:

Sadly, most laptops rarely survive 1 year under the best conditions

Wat.

 

That seems extremely low -- and our laptops are also 8-9 hours a day, 5 days a week.

[Kisai]

2 hours ago, shadow_ray said:

What do you mean by "survive"? If it only suffers from a minor issue do you count it as dead? One year is a really short period of time.

 

Why they keep the machines running? I really don't understand. Mine goes to sleep after 30 mins or so by default.

Because they replaced all the office desktops with laptops and never changed the power settings from high performance, and they supposedly did that so that they would stay powered to download nightly updates. 

 

[hishnash]

On 7/6/2022 at 9:56 AM, colonel_mortis said:

I thought the chip shortage was supposed to be mostly resolved at this point, and I would have imagined that a company with the purchasing power of Microsoft would be one of the last to be affected.

Not at all, lots of legacy nodes, and other additional chips are still in very high demand, stuff like power stage etc. If you consider a server with 16+ memory slots that is a LOT of power management systems. 

 

While yes MS could pay for this there is no point doing so if it ends up costing so much that you cant get return on your investment or you end up having way to many different skews since you just cant (regardless of money) secure enough of a single part to build 10000 units. Since they need to go through some extreme validation of anything new just changing a single power stage vendor or model number will end up costing them a lot in $ and time validating the system works as expected.   This is not a gaming rig were they just build it then run it and find out later if its a little prone to crashing they test and test and test again an again for any tiny change before rolling things out in production as what will cost them the most is if they skew up with some big customer who then leaves MS to a different vendor due to them having a few failing servers. 

[AlTech]

On 7/6/2022 at 4:50 PM, LAwLz said:

A few months ago I had someone tell me, with a straight face, that the shortages were caused by crypto miners and that no other industry except gaming GPUs were affected...

 

Anyone who is slightly involved with the industry (and no, buying graphics cards to play games is not being involved) could have told you that we were heading this way months ago. Lead times on new severers have not exactly going down. 

Want a new sever? You better order it today if you want it next year. Even servers without GPUs. 

 

Shipping issues. 

Lockdowns. 

Material shortages. 

Restrictions on imports/exports of things like noble gas from Russia. 

High inflation. 

High demand. 

 

All factors of why things are the way they are. 

 

 

At this point I just hope prices will return to somewhat normal during 2023.

In the case of the UK specifically there's also trade barriers between the UK and the EU and between the UK and the rest of the world which is leading to increased delays for getting new things (chips and products using chips included) and higher costs when buying them.

[Lurick]

On 7/7/2022 at 5:48 PM, LAwLz said:

One of my clients makes controller boards for fairly niche industrial equipment. 

Because of the drastic shortages for things like certain IC chips some staff members have been reassigned from assembling, to just looking at sites like eBay after components.

Then if they find some component they need they order it from whichever seller they can, usually at like 5 to 10 times higher than the usual price. 

 

It gives pretty good insight into what a massive domino effect certain things have. 

A factory in China shut down because of covid.

This lead to a certain IC being out of stock.

Now a company can't build a controller board because of that missing IC.

That in turn means that a different company can't build their forklift which uses they controller board. 

That means that a third company are now several forklifts short and are not operating at 100% efficiency. 

 

And the spiral goes on... 

And that's just one company. The same is happening to thousands of companies right now, with tens of thousands of components being in short supply and high demand. 

Yup, hell there are some super specific power cables I need and they have around a 200 day lead time, it's crazy right now.

[DANK_AS_gay]

How did people ever think "just-in-time" shipping was a good idea. This is incredibly stupid. 

Russia is seriously affecting far more than I ever thought they could. Everyone knows this. America did the "Cash-for-Clunkers" program to help automakers, why can't they do a "Fuck Putin" program to help automakers. 

I mean, this also is affecting Microsoft, and IBM, AMD, nVidia, Intel, Matrox (yes, they still exist), and whatever else less prominent chip makers that make legacy parts. And it made getting GPUs for our armed forces (A100s for an Air Force supercomputer were delayed by 8 months) difficult and expensive. Etc. Etc. This affects everyone. This is also the one time in like 100 years America can fight in a war and be the good guys. I'm saying this as a 17 year old relatively fit (i.e, not overweight/obese) male that would be drafted if the war went on for over 8 months. I would be willing to fight over this.

[ToboRobot]

I didn't see any quotes from informed parties that this is chip shortage, and not a space, cooling, or power limit.

But I don't think this is abnormal, I think perhaps the last few years have been the abnormal over production.

[shadow_ray]

1 hour ago, DANK_AS_gay said:

How did people ever think "just-in-time" shipping was a good idea. This is incredibly stupid. 

No it's not stupid. It's cheaper and more effective because resources are not wasted as much if timing is right.

 

1 hour ago, DANK_AS_gay said:

This is also the one time in like 100 years America can fight in a war and be the good guys. I'm saying this as a 17 year old relatively fit (i.e, not overweight/obese) male that would be drafted if the war went on for over 8 months. I would be willing to fight over this.

I don't think real life is that black and white. The world is a big chess game where the big players sacrifice pieces for their own benefit, and you are willing to become a pawn over some inconveniences. Colors are just for distinctions, motives are often similar.

[leadeater]

1 hour ago, shadow_ray said:

No it's not stupid. It's cheaper and more effective because resources are not wasted as much if timing is right.

The problem is also bad enough for long enough it makes no difference either way. Zero supply is zero supply, that's from manufacturing shutdowns and reduced output when actually working. And then there is raw materials shortages too, for the same/similar reasons.

[leadeater]

3 hours ago, ToboRobot said:

I didn't see any quotes from informed parties that this is chip shortage, and not a space, cooling, or power limit.

But I don't think this is abnormal, I think perhaps the last few years have been the abnormal over production.

If you are a regular server purchaser, like myself, then you'll know this is a supply issue. Azure datacenter facilities are built large enough they can expand within them for at least 5 years and the site itself can have more buildings built etc. If a hold on deploying resources is put in place it's because of server and networking equipment not space, cooling or power.

 

2020 and backwards I could CTO any server, literally any configuration, and it'll be built in Singapore and shipped to NZ and in my rack within 60 days. Post 2020 90 days is damn good aka lucky, up to 180 normal, being told you can't have that part and must change it also normal.

 

I don't get to choose what I want anymore, like specific NIC, I get told this one, only this one is available. Choice is dead. Just try and get a Broadcom 25Gb OCP 3 NIC, only way is mission impossible style in to someone's datacenter who already has some from 2020.

[DANK_AS_gay]

6 hours ago, shadow_ray said:

not wasted as much if timing is right.

You can never ensure that there will not be a delay. Having 0 buffer in case of a truck accident or a train derailing is idiotic. That leads to inefficiency and delays if everything doesn't go 100% perfectly, which literally never happens. Relying on global relations to remain the way they are is also idiotic. You can never ensure timing is right, meaning that every week is a gamble. For what? Not taking care of a warehouse in the midwest? Was the cost of said warehouse really worth the risk?

7 hours ago, shadow_ray said:

I don't think real life is that black and white.

WW2 was the greatest thing to happen to the US. It was a largely black and white affair (ignoring the Japanese camps). It made being proud of the US easy. Ukraine v Russia is the easiest to distinguish I've ever seen, and certainly is more complicated than it may seem at first glance, but to the US, it means a bit more than chip shortages. Remember when the US was an English colony? Well, no, you aren't 200 years old, but anyways... It was a bunch of backwater hicks with no formal military, weapons training, no Navy, vs the biggest empire the world has seen. We won. David vs Goliath. Ukraine vs Russia. Ukraine's situation is somewhat different, but the major factor, the Big vs Little, is what Americans understand. This is the first chance in years for us to start to repair our global image. To show that we aren't hypocrites. And to me, I'm sick of sitting wringing my hands worrying. I want to DO something.

I'm also sick of making myself do things like eat when my ADHD makes self-motivation nigh-on impossible.