navi 21 AMD Radeon RX 6000 GPUs Mysteriously Start Dying, German Repair Shop Receives 48 Cards With Cracked Chips - Is it from a Driver Update? (Updated)

[BiG StroOnZ]

Summary

There are now reports of the older Radeon RX 6000 GPUs which have mysteriously started dying. German Techtuber and repair shop, KrisFix-Germany, posted a video sharing how his shop has received several AMD Radeon RX 6000 series graphics cards for repair. The shop got 61 cards in total which were a mix of Radeon RX 6900 and RX 6800 series flavors and out of 61, 48 cards had come with a dead GPU. Kris states that this is the first time he has seen so many GPUs come to his shop with dead GPUs. He mentions on his Facebook, "are AMD cards dying after a driver update?"

 

 

 

 

 

Quotes

Quote

It's not just the GPU that has died, all 48 cards come with shorted SOC rail, shorted memory rail and shorted memory controller rail. Upon asking users what they were specifically doing with the cards, most of them responded differently. Some said they were gaming normally while others were watching YouTube or even had their PC on idle when the GPU died on them. But one thing that matched across all affected GPUs was the driver version. It is stated that all GPUs were using the latest drivers which launched last year in December aka Adrenalin 22.11.2 (WHQL).

 

For now, we can advise users to not use the latest drivers for their cards as Buildzoid states that it won't be the first time that a GPU manufacturer would accidentally disable the thermal protection in a new driver release. Some users have also reported hearing aggressive coil whine on the newer drivers which wasn't there before.

 

My thoughts

This definitely sucks, as it seems now AMD is going to have to deal with two GPU related issues; both this gen and last gen. AMD has not yet fully resolved the 7900 XTX MBA RMA issue, and I recently updated that thread showing that AMD is running out of GPU replacements. Which possibly means there are way more affected GPUs out in the wild than AMD proclaims. Obviously, further investigation needs to be done here before specifically blaming a design flaw or a driver. However, as it stands, 48 cards with the same issue seems very peculiar. Buildzoid seems to agree that it wouldn't be the first time that thermal protections were disabled through a driver, and I concur with this. We've seen this before, even with NVIDIA. Currently, it seems that this affects Navi 21, and Navi 22 and Navi 23 are not mentioned as being affected ATM, AFAICT.  Hopefully the community gets to the bottom of this, and I will update the thread as more information develops.

 

Sources

https://wccftech.com/amd-radeon-rx-6000-gpus-mysteriously-start-dying-german-repair-shop-receives-48-cards-with-cracked-chips/

 

Update

 

Summary

KrisFix provides an update to an unusual increase of Radeon RX 6000 GPU failures at their repair center. It’s not the drivers, but crypto mining and air humidity.

 

 

Quotes

Quote

The long story short is that it was not the driver after all. As it turns out, the graphics cards that were sent over to KrisFix also had one other thing in common that was previously not mentioned. Many of these cards were sold by the same seller, with a strong suspicion that these cards were previously used for cryptomining.

 

However, the main issue was not the fact they were used for mining, as many post-mining cards are used by gamers around the globe every day. It is the humid condition that these cards were kept in, that might have been the cause of the problem. Prolonged storage of graphics cards in a warehouse with high moisture is a very plausible explanation for GPU cracking.

 

Such conclusions were shared after conducting 150 hours of tests with multiple cards, including AMD MBA (Made by AMD) and custom designs. These tests did not provide many answers, but a more detailed survey among customers definitely did.

 

My thoughts

It seems since this was a localized incident that the final conclusion makes sense. While there still are some questions remain unanswered. It seems pretty obvious that it was not a driver issue. Which as mentioned by many, would have had this issue pop up from more than one location, rather than solely Germany. 

 

Sources

https://videocardz.com/newz/radeon-gpu-cracking-not-caused-by-drivers-storing-conditions-and-cryptomining-to-blame

[SeriousDad69]

When it rains it pours I guess. Stuff like this is why I don't even consider buying Radeon unless they're 30-40% cheaper than the equivalent Nvidia card. This is 100% anecdotal with a tiny sample size, but every AMD card I've had has died. I've had an HD 7970, R9 270, RX Fury, and they all died within 1-2 years of the warranty ending. I have GTX 900 and 1000 cards that I've given out to younger people in my family for their first builds and literally all of them are still working.

Cant even remember the last time Nvidia pushed out a driver that killed cards, must've been the GTX 400 series or older.. But yeah this might not be driver related, it might not even be AMD's fault. We need more information and investigations by Buildzoid, Gamers Nexus, Igor, Derbauer, etc.

[tim0901]

This is certainly concerning, but as you said it needs proper investigation before people start throwing blame and panicking.

 

Definitely not what AMD needs right now though. They've only just managed to get rid of their reputation of having terrible GPU drivers - and I'm not sure how much of that is because Intel's are just so much worse. The last thing they need is a new reputation that their GPUs aren't reliable and get bricked with driver updates.

[WereCat]

"But one thing that matched across all affected GPUs was the driver version. It is stated that all GPUs were using the latest drivers which launched last year in December aka Adrenalin 22.11.2 (WHQL)." 

 

 If driver was the issue there would be most likely many, many more cards affected and you would hear about it from across the world and not just single repair shop in Germany. 

 

We don't know what kind of abuse these cards went trough. 

Were they taken apart by the user in the past? 

etc... 

 

[SeriousDad69]

47 minutes ago, WereCat said:

"But one thing that matched across all affected GPUs was the driver version. It is stated that all GPUs were using the latest drivers which launched last year in December aka Adrenalin 22.11.2 (WHQL)." 

 

 If driver was the issue there would be most likely many, many more cards affected and you would hear about it from across the world and not just single repair shop in Germany. 

 

We don't know what kind of abuse these cards went trough. 

Were they taken apart by the user in the past? 

etc... 

 

It might be happening to a lot of cards, but most(maybe all?) 6800XTs and 6900XTs are still under warranty, so most users will just curse their luck and send it in for RMA. Take this with a pound of salt, but I read somewhere that Germany has different laws than the US regarding warranties that might give more of an incentive to send electronics to a third party repair shop instead of trying for an RMA.

Looking at this channel it seems like this guy repairs a lot of user serviced cards(replaced thermal paste, new pads, etc) so maybe they cant or don't think they can get an RMA, and that revealed the issue. (Now that I'm thinking about it, I wonder just how many glaring product defects are covered up by good warranty practices......)

[BlueChinchillaEatingDorito]

Hard to wrap my head around how a driver update can cause the actual die to crack or shatter like that. 

 

37 minutes ago, SeriousDad69 said:

Looking at this channel it seems like this guy repairs a lot of user serviced cards(replaced thermal paste, new pads, etc) so maybe they cant or don't think they can get an RMA, and that revealed the issue. Now that I think about it, I wonder just how many glaring product defects are covered up by good warranty practices.

If that's the case, could this just be an instance of user error at a large scale? Not sure. As someone on Twitter has mentioned, the older folks in the audience would remember the dangers of over torquing coolers and cracking the CPU die on those early Athlon and Pentium 3 chips. But that wouldn't explain why it's only AMD cards so far that seems to be a trend as it's not like Nvidia cards have heat spreaders. Nor was the damage back then that spectacular. Usually it would've just been chipped edges of the CPU die. Still rendered them useless but it's not like the whole thing shattered like glass. 

[SeriousDad69]

20 minutes ago, BlueChinchillaEatingDorito said:

If that's the case, could this just be an instance of user error at a large scale? Not sure.

I don't think so. It would've been a constant trickle since the cards came out, and like you said it would've been all models including Nvidia. Having a sudden dump of 50+ busted cards could just be AMD getting unlucky, but I don't really believe in luck that much.

[BlueChinchillaEatingDorito]

6 minutes ago, SeriousDad69 said:

I don't think so. It would've been a constant trickle since the cards came out, and like you said it would've been all models including Nvidia. Having a sudden dump of 50+ busted cards could just be AMD getting unlucky, but I don't really believe in luck that much.

And if it's 50 cards from all sorts of board partners, that'll even more worrying as you can't say it's maybe a cooler design/manufacturing defect causing the damage. We don't know if it's specifically one vendor or reference cards. 

[Brooksie359]

7 minutes ago, SeriousDad69 said:

I don't think so. It would've been a constant trickle since the cards came out, and like you said it would've been all models including Nvidia. Having a sudden dump of 50+ busted cards could just be AMD getting unlucky, but I don't really believe in luck that much.

But you would also see this at more than a single shop. If it's only one place reporting it then it seems like an isolated incident as again we would see this around the world if it was something fundamentally wring with the cards. 

[BlueChinchillaEatingDorito]

12 minutes ago, Brooksie359 said:

But you would also see this at more than a single shop. If it's only one place reporting it then it seems like an isolated incident as again we would see this around the world if it was something fundamentally wring with the cards. 

That too. Especially if they're questioning a driver update which everyone around the globe gets at the same time... which in of itself I'm still perplexed how one comes to that conclusion. 

[Chiyawa]

32 minutes ago, Brooksie359 said:

But you would also see this at more than a single shop. If it's only one place reporting it then it seems like an isolated incident as again we would see this around the world if it was something fundamentally wring with the cards. 

Could be Windows silently updating the system with the driver while people are gaming, which cause hard lock and hard crash. This, couple with the chip is way too hot could potentially make the silicon under a lot of thermal stress which crack the chip, but I doubt it could be that issue...

 

But come to think of it, could this sucker (not the shop, I mean the so call 'victim(s)') mine with their GPU card?

[starsmine]

I am beyond suspicious of the claim of a driver update physically cracking the chips. That is just wild and I would need SIGNFICANT Proof to give it any credit.

[xAcid9]

Are these 2nd hand mining cards? 

[BlueChinchillaEatingDorito]

1 hour ago, Chiyawa said:

Could be Windows silently updating the system with the driver while people are gaming, which cause hard lock and hard crash. This, couple with the chip is way too hot could potentially make the silicon under a lot of thermal stress which crack the chip, but I doubt it could be that issue...

 

But come to think of it, could this sucker (not the shop, I mean the so call 'victim(s)') mine with their GPU card?

But how hot would it need to get in order to shatter the chip like that. Shouldn't GPUs shutdown like CPUs if it goes beyond a certain threshold? And not thermally runaway until it melts?

[Kisai]

3 hours ago, SeriousDad69 said:

When it rains it pours I guess. Stuff like this is why I don't even consider buying Radeon unless they're 30-40% cheaper than the equivalent Nvidia card. This is 100% anecdotal with a tiny sample size, but every AMD card I've had has died. I've had an HD 7970, R9 270, RX Fury, and they all died within 1-2 years of the warranty ending. I have GTX 900 and 1000 cards that I've given out to younger people in my family for their first builds and literally all of them are still working.

Cant even remember the last time Nvidia pushed out a driver that killed cards, must've been the GTX 400 series or older.. But yeah this might not be driver related, it might not even be AMD's fault. We need more information and investigations by Buildzoid, Gamers Nexus, Igor, Derbauer, etc.

The last few AMD cards I had, had "slow failures", but that was ages ago (like HD7750-ago.) I only ended up switching to NVIDIA through a combination of "someone gave the GTX 760 to me" and subsequent nvidia cards were just more available.

 

But in terms of clients's computers, AMD parts only ever showed up in the HP machines, and they would accept either the Radeon or FirePro drivers, which lead to some interesting configurations that were "technically working" but were not actually the right drivers.

[Chiyawa]

1 minute ago, BlueChinchillaEatingDorito said:

But how hot would it need to get in order to shatter the chip like that. Shouldn't GPUs shutdown like CPUs if it goes beyond a certain threshold? And not thermally runaway until it melts?

That's true. Though it can be cause by thermal shock, which the silicon all of a sudden gets too hot (we're talking from a low temperature to a very high temperature in a very short time), like when you pour boiling water into a glass of ice. If that's the case, the GPU might not have enough time to react. Still, this scenario is very unlikely, but it could happen if over-volted to the max.

[SteveGrabowski0]

8 hours ago, SeriousDad69 said:

When it rains it pours I guess. Stuff like this is why I don't even consider buying Radeon unless they're 30-40% cheaper than the equivalent Nvidia card. This is 100% anecdotal with a tiny sample size, but every AMD card I've had has died. I've had an HD 7970, R9 270, RX Fury, and they all died within 1-2 years of the warranty ending. I have GTX 900 and 1000 cards that I've given out to younger people in my family for their first builds and literally all of them are still working.

Cant even remember the last time Nvidia pushed out a driver that killed cards, must've been the GTX 400 series or older.. But yeah this might not be driver related, it might not even be AMD's fault. We need more information and investigations by Buildzoid, Gamers Nexus, Igor, Derbauer, etc.

I had my GTX 970 die on me, but I can't complain because it was after more than five years of heavy usage and that was an awesome card... even though it was about the worst overclocking 970 on this entire forum haha. Hope I have better luck than you since I just got a 6700 XT a month and a half ago (first AMD card I have used, though I had an ATi Radeon 8500 back from 2001-06 that was awesome).

[leadeater]

9 hours ago, BlueChinchillaEatingDorito said:

But how hot would it need to get in order to shatter the chip like that. Shouldn't GPUs shutdown like CPUs if it goes beyond a certain threshold? And not thermally runaway until it melts?

I did a bad temporary mount of my air cooler on one of my 290X's that was water cooled, needed to run it in different system. Couldn't find the original screw, blah blah tl;dr the cooler wasn't secure like I thought it was and it popped off the GPU die while under a gaming load, the die did not crack.

[Ripred]

hmmm, my 5600xt all up to date and hasn't exploded yet, I'm not particularly nice to it either, none of these people had a sensor log before meltdown? more information is needed, lest we forget the New world 3090 incident

[porina]

The reports of everyone using the same driver doesn't immediately strike as something high on the list to worry about. People generally keep up to date with drivers, and being on reportedly the latest one is expected.

[WereCat]

2 hours ago, porina said:

The reports of everyone using the same driver doesn't immediately strike as something high on the list to worry about. People generally keep up to date with drivers, and being on reportedly the latest one is expected.

It's just a confirmation bias. You could easily say that all users with broken cards were sitting when it happened so it's recommended for people to stand when using the PC. 

[Jerakl]

I actually had a friends 6000 series die seemingly at random (2 years of ownership, light gaming usage). His hadn't completely died, but it DID get awfully funky and he upgraded drivers (to no avail) on my recommendation. It doesn't seem like his issues were driver related but if this is true that certainly didn't help (as it usually does). 

17 hours ago, SeriousDad69 said:

When it rains it pours I guess. Stuff like this is why I don't even consider buying Radeon unless they're 30-40% cheaper than the equivalent Nvidia card. This is 100% anecdotal with a tiny sample size, but every AMD card I've had has died. I've had an HD 7970, R9 270, RX Fury, and they all died within 1-2 years of the warranty ending. I have GTX 900 and 1000 cards that I've given out to younger people in my family for their first builds and literally all of them are still working.

Cant even remember the last time Nvidia pushed out a driver that killed cards, must've been the GTX 400 series or older.. But yeah this might not be driver related, it might not even be AMD's fault. We need more information and investigations by Buildzoid, Gamers Nexus, Igor, Derbauer, etc.

 

I haven't owned an amd gpu since I was given hand-me-down ATI 4770s or something.

 

My 980 ti (that I bought USED) still works perfectly and is in my gfs pc. I ran that card pretty hard for a while and even had to replace the fans on it (bearings on the central fan went bad), but the actual processor itself is still a-OK. This gpu is now like 6-7 years old? 

 

As far as Nvidia putting out a driver that killed cards I can't recall it happening anytime in recent memory, but every now and then they do let a botched one get out that has adverse affects. 

 

 

4 hours ago, porina said:

The reports of everyone using the same driver doesn't immediately strike as something high on the list to worry about. People generally keep up to date with drivers, and being on reportedly the latest one is expected.

Also as pc enthusiasts most keep drivers up to date, so it only follows that most/all would be running the latest bios. So while there could be something to it assuming it's bc of that kinda falls flat imo

[porina]

47 minutes ago, Jerakl said:

Also as pc enthusiasts most keep drivers up to date, so it only follows that most/all would be running the latest bios. So while there could be something to it assuming it's bc of that kinda falls flat imo

I'd guess most will allow driver updates and actually update them shortly after release. However I don't see BIOS updates the same way. I think I've only needed to do it once ever on my early 3070 to have the ReBAR support added. I had to go out of my way to find the software needed to update that as it required the AIB's software which I don't normally use.

 

Before that were a couple of minor patches, I forget exactly what for, but I think one was DP compatibility around Pascal era. Those were direct from nvidia but you had to go get them yourself. It wasn't part of a driver update.

[Montana One-Six]

20 hours ago, SeriousDad69 said:

Take this with a pound of salt, but I read somewhere that Germany has different laws than the US regarding warranties that might give more of an incentive to send electronics to a third party repair shop instead of trying for an RMA.

Generally Germany has better consumer protection laws than the US. Manufacturers in Germany are required by law to provide a 2 year warranty period (at least) for new electronics. That warranty period begins with the purchase meaning first day of the warranty is the date of purchase.

 

Since the RX 6900 and 6800 came out a bit over 2 years ago, the warranty period for these cards if bought the first few weeks has expired by now. Which could explain the sudden dump of damaged cards because people try to get them fixed by some third party instead of going the warranty route since it is no longer an option.

[YellowJersey]

On 1/11/2023 at 6:08 PM, BlueChinchillaEatingDorito said:

And if it's 50 cards from all sorts of board partners, that'll even more worrying as you can't say it's maybe a cooler design/manufacturing defect causing the damage. We don't know if it's specifically one vendor or reference cards. 

Yeah, that's the part that I'm curious about: is this one vendor, reference cards, or a mix of the two?

Edit: About 30 seconds after I made that comment, the video JayzTwoCents I'm watching says that it's board partners and reference cards. (around 7:08)