Intel says their 13th/14th Gen instabilities are from 'elevated operating voltages stemming from a microcode algorithm'

[Skiiwee29]

Intel has released an official statement regarding this. It can be found here:

 

https://community.intel.com/t5/Processors/July-2024-Update-on-Instability-Reports-on-Intel-Core-13th-and/m-p/1617113#M74792

[Bitter]

I'll hold my breathe, I'm interested in the line of research looking at oxidation of the TaN layer causing high resistance on the copper vias leading to increased voltage and heat.

[Somerandomtechyboi]

2 hours ago, vanpuffelen said:

My gut thinks this TVB thingamabob is related to my old i7-3770K automatically getting OC'd to 3.90GHz, even though it had a shoddy Intel stock cooler connected to it. 

thats ancient ivybridge so no tvb just regular turboboost, afaik tvb was first introduced with 10th gen

 

3 hours ago, Karthanon said:

Intel says the issue is from elevated operating voltages due to microcode, so microcode fixes shall kill the problem

too conservative with the voltages they run so they run too much voltage for no reason so higher power draw (battery life and performance goes to shit on laptops) alongside higher temps which goes hand in hand with degradation, 1.5v at 70c isnt the same as 1.5v at 100c and this only gets exponentially worse the higher the voltage goes 

 

Spoiler

heres an example of setting an overly conservative voltage when a lower voltage wold work fine for a given setting with the pic above and the pic below is what these gdies will actually run at the same voltage (2.3v) and it probably only needs ~2v maybe even less for 2800c11

 

only reason its not erroring out due to overvolt is because i put a fan over them (still got hot to the touch but 40-50c is cool enough) and the ics are pretty tolerant towards sloppy voltage settings, didnt really intend to run at c11 just forgot to set it to c10 when i loaded a profile and only realized it after i stopped p95 after 12 hours

 

 

intel is running voltages that are too conservative and theyre unneccesarily boosting their chips too high and also running too hot so ofc the things are gonna degrade with that combo of high volt + high temp + weak silicon that cant handle high volt

 

3 hours ago, SpookyCitrus said:

@Karthanon @podkall @LinusTech

 

So my 14900K is one of the affected units, I upgraded from a 12700K to it a month after release. It was fine for three months then whamo bamo, Cinebench no longer running, DX12 issues with my 4090 being "out of video memory", etc. I used the Intel XTU fix to lower the performance core ratio from 57x to 55x and it works just fine. I have only had issues when for some reason XTU stops auto applying my settings on start up when it needs an update. Would it be wise to RMA my CPU while I still have the option or wait to see if this fix, when it comes out, alleviates the issue first? To be clear mine is not one that has gotten worse over time like some others who have posted about this issue. 

3 months is kinda ridicolous though should definitely rma and when you get your next one probably a good idea to undervolt it around its best v/f (ex 5.5g 1.3v vs 5.3g 1.2v ofc go for the latter) alongside disable intel cep, lower temps volts powerbills and you wont take much of a performance hit either since these are already being pushed well into dminishing returns for v/f

 

if theyre really gonna push these that hard could have atleast made the silicon more voltage resistent

[Tetras]

Intel have confirmed on reddit that the leak about a manufacturing issue was accurate, however they also claim that very few of the returned CPUs had instability due to that problem and most were due to the overvolting.

 

They have also clarified (to THG, I believe) that the damage is indeed permanent and the microcode update will not fix those CPUs.

 

 

[dysorder]

Are they inferring that undervolting would fix this? If so, how much?

[Likwid]

9 hours ago, porina said:

Random thoughts:

Voltage demands can vary a lot, so it doesn't necessarily mean the wrong voltage was present all the time.

Depending on the magnitude of error and how often it happened, it does sound like it could cause degradation.

Mobo makers pushing beyond spec certainly wouldn't have helped at all, so that still needed to be addressed regardless.

Have all microcode for affected CPUs been affected since release, or was it introduced with a mid-lifespan update?

 

Presuming the '900 parts are most affected since they're sailing closest to the wind. '700 and lower get progressively less likely to be affected as they might have a bit more margin to play with.

For me, it seems like Intel made this a problem, with new power profiles you get a 9% performance reduction compared to what they advertised.
And what about non-OC motherboards where CPUs are also failing?

Let's look with a different perspective, that CPUs that fail are degraded as "partially failed". When you OC your GPU for example you increase clock speeds and then get crashes, what do you do next? You increase the voltage until you don't crash anymore. So what happens when increasing voltage doesn't help? You reduce the clock speed so it won't crash anymore, and the result is that your overclocked part has degraded (became lower quality silicon, as higher quality gets higher clocks with less voltage) and will continue to degrade until it won't be usable if you run it with maximum performance.

TLDR: Damage is permanent and only reduced performance (lower than stock) can fix this issue.
 

[porina]

8 minutes ago, Likwid said:

For me, it seems like Intel made this a problem, with new power profiles you get a 9% performance reduction compared to what they advertised.

What exactly did Intel advertise? Do not confuse it with what everyone else in the tech media said. Turbo has always been opportunistic and not a guaranteed sustained performance level.

 

8 minutes ago, Likwid said:

And what about non-OC motherboards where CPUs are also failing?

It's the same CPUs with same microcode, so have the same risk.

 

I've said it before where I suspected this is not one problem, but could be many getting grouped together as awareness increased. We now know of:

1, oxidation issue for some early 13th gen. Should be easily confirmed by dates/serial number.

2, microcode error - to be worked through.

3, excessive mobo settings - a long term problem stretching beyond a decade, and hopefully mobo defaults will be more responsible going forwards.

 

Intel will help you if you're affected by 1 or 2.

 

8 minutes ago, Likwid said:

TLDR: Damage is permanent and only reduced performance (lower than stock) can fix this issue.

If you're confirmed affected then get warranty replacement. The update is to prevent still working CPUs from going bad and not a universal fix for already unstable CPUs.

[Glazarus]

Can someone confirm that the latest Asus bios (personally using the proart motherboard) update is stable enough? I rarely update bios on a working system, but I still have a working i9 13th gen so right now I’m afraid to run my PC, but also not sure if I should update until I know for certain that there’s no other issues with the latest Asus bios. Asus was my favorite brand but right at the time I bought this system they got criticized so we are living in times in which all those stable reliable brands you chose before seem to have lost all credibility and we can just hope things don’t crash and burn. 
 

A fix here might break something else over there, no one seems to know. And people wonder why so many love Apple so much. With the cloudstrike thing and this, how the hell can you even plan a build for stability and critical work? It’s ridiculous.

[Tetras]

6 hours ago, Glazarus said:

Can someone confirm that the latest Asus bios (personally using the proart motherboard) update is stable enough? I rarely update bios on a working system, but I still have a working i9 13th gen so right now I’m afraid to run my PC, but also not sure if I should update until I know for certain that there’s no other issues with the latest Asus bios. Asus was my favorite brand but right at the time I bought this system they got criticized so we are living in times in which all those stable reliable brands you chose before seem to have lost all credibility and we can just hope things don’t crash and burn. 
 

A fix here might break something else over there, no one seems to know. And people wonder why so many love Apple so much. With the cloudstrike thing and this, how the hell can you even plan a build for stability and critical work? It’s ridiculous.

In the reddit thread where Intel confirmed the manufacturing issue, the Intel rep strongly recommended in the comments that you keep your BIOS updated while they work through the issue. I take that as meaning even if you don't have a BIOS intended to fix this/any particular problem, it may still be very important for things they have done "behind the scenes".

 

Would I personally update if my PC was working perfectly fine? I'm not sure..., the idea that the bug could be degrading my CPU is a worrisome one.

 

Since your board is a workstation board, does your board not use very conservative settings for your CPU anyway?

[SpookyCitrus]

Kind of ranty, sorry in advance for the word vomit.

 

Little update. I talked with Intel today. Was told by two separate reps that the microcode patch(according to them) should fix the instability with my 14900K, I asked multiple times, both reps, if it was going to be necessary to RMA the CPU and both said no, even after clarifying it was already "technically" degraded. Though the second rep did add that while the new update coming in August should fix the issue, if my processor has issues with stability after the update that I would need to reopen my ticket.

 

I also talked with Gigabyte afterwards as the Intel rep was unsure if my Z690 board would receive the bios update to fix the issue. And I'm being a spaz worrying that it won't as it hasn't received any updates since December including the CEP patch Gigabyte released not too long ago for the Z790 boards(in my research I couldn't find why the Z690 boards did not receive this, my only guess is they don't use that particular feature as there are no CEP options in my bios settings) or any other updates. The rep from Gigabyte said that all boards affected will be getting the update, when it is ready. Though they did not go into any detail regarding my board or Z690 in general. I figure it's a safe bet it will receive the update. I get it's entirely to early for them to give a definitive answer, I'm sure it takes time to prepare and test this type of thing.

 

Prior to hearing back from Gigabyte I decided on and ordered a Z790 board and DDR5 memory, it was on sale and not terribly priced anyway and I figured better safe than sorry. Since receiving the response from Gigabyte I am now on the fence with whether or not I should continue with replacing my current board and ram with the Z790 and DDR5 I have ordered, or return it once it has arrived(it was too late to cancel by the time that I had received the response from Gigabyte). I ordered the parts from Best Buy and it's not a hassle for me to return them to my local store once they arrive. But I am still slightly worried that my Z690 board may not be the best home for my 14900K and do not know when the update will go through, I'd prefer to be able to have this issue resolved asap, over waiting for an update to come to Z690 if it is delayed longer than Z790.

 

I also contemplated switching to AMD for most of the day as it's been a pretty annoying situation to deal with, however, I've dealt with other headaches on the AM5 systems I've worked on in my shop and have had some major grievances in the past with personal AMD rigs. I thought about purchasing an X670 board, DDR5, and a 7800X3D or 7950X3D then selling my Z690 board and DDR4 ram kits, RMAing the 14900K then selling that as well. The AMD route was more expensive initially but has way more profitability with selling my old parts, there is also the added bonus of it having at least one more generation of processors being released for it. I decided to stick with the 14900K as the Z790 board and ram are cheaper on initial cost and I should be able to recoup if not all, most of the money spent on the board and ram by selling my current board and the few DDR4 kits I no longer need, sticking with Intel will also help alleviate some of the headache with drivers, as well as most of the software I use for my side job workload has some features that prefer Intel processors. 

 

I'm curious to see if the reps were  justified in their confidence regarding not having to RMA my processor, as the first one I was disconnected from for technical difficulties on their end wasn't even aware of the update from yesterday regarding the microcode error. A follow up was sent by the first Intel rep after the chat was ended asking if they can close the ticket, again, reiterating that I should not have to RMA my processor. Although they did add that the second rep did advise me to monitor my processor after the update and if it has further issues contact support.

 

Most people outside of Intel are skeptical and have been pretty sure that degraded processors will need to be RMA'd. I myself am included in thinking that an overvolted processor that has degraded and is unable to function at stock settings would need to be RMA'd. I mean a software update can't fix physical damage caused by the chip being overvolted.

 

Curious if anyone else was in my shoes. What you would do? Return the Z790 and DDR5 ram and just hope for a quick update for my Z690 board, return them and go for a platform swap to AMD? Or just keep the Z790 and DDR5 ram? I am aware that I most likely jumped the gun with purchasing the new board and ram, I've just been kind of frustrated with the whole situation and having to deal with it since my processor initially started having the issue.

[Poinkachu]

 

I think it's quite funny that the Oxidation problem confirmation is just at Reddit :x

[RejZoR]

Crank everything up to eleven to get amazing results in reviews and then patch it down so it even works.

[Bitter]

6 hours ago, SpookyCitrus said:

Kind of ranty, sorry in advance for the word vomit.

 

Little update. I talked with Intel today. Was told by two separate reps that the microcode patch(according to them) should fix the instability with my 14900K, I asked multiple times, both reps, if it was going to be necessary to RMA the CPU and both said no, even after clarifying it was already "technically" degraded. Though the second rep did add that while the new update coming in August should fix the issue, if my processor has issues with stability after the update that I would need to reopen my ticket.

 

I also talked with Gigabyte afterwards as the Intel rep was unsure if my Z690 board would receive the bios update to fix the issue. And I'm being a spaz worrying that it won't as it hasn't received any updates since December including the CEP patch Gigabyte released not too long ago for the Z790 boards(in my research I couldn't find why the Z690 boards did not receive this, my only guess is they don't use that particular feature as there are no CEP options in my bios settings) or any other updates. The rep from Gigabyte said that all boards affected will be getting the update, when it is ready. Though they did not go into any detail regarding my board or Z690 in general. I figure it's a safe bet it will receive the update. I get it's entirely to early for them to give a definitive answer, I'm sure it takes time to prepare and test this type of thing.

 

Prior to hearing back from Gigabyte I decided on and ordered a Z790 board and DDR5 memory, it was on sale and not terribly priced anyway and I figured better safe than sorry. Since receiving the response from Gigabyte I am now on the fence with whether or not I should continue with replacing my current board and ram with the Z790 and DDR5 I have ordered, or return it once it has arrived(it was too late to cancel by the time that I had received the response from Gigabyte). I ordered the parts from Best Buy and it's not a hassle for me to return them to my local store once they arrive. But I am still slightly worried that my Z690 board may not be the best home for my 14900K and do not know when the update will go through, I'd prefer to be able to have this issue resolved asap, over waiting for an update to come to Z690 if it is delayed longer than Z790.

 

I also contemplated switching to AMD for most of the day as it's been a pretty annoying situation to deal with, however, I've dealt with other headaches on the AM5 systems I've worked on in my shop and have had some major grievances in the past with personal AMD rigs. I thought about purchasing an X670 board, DDR5, and a 7800X3D or 7950X3D then selling my Z690 board and DDR4 ram kits, RMAing the 14900K then selling that as well. The AMD route was more expensive initially but has way more profitability with selling my old parts, there is also the added bonus of it having at least one more generation of processors being released for it. I decided to stick with the 14900K as the Z790 board and ram are cheaper on initial cost and I should be able to recoup if not all, most of the money spent on the board and ram by selling my current board and the few DDR4 kits I no longer need, sticking with Intel will also help alleviate some of the headache with drivers, as well as most of the software I use for my side job workload has some features that prefer Intel processors. 

 

I'm curious to see if the reps were  justified in their confidence regarding not having to RMA my processor, as the first one I was disconnected from for technical difficulties on their end wasn't even aware of the update from yesterday regarding the microcode error. A follow up was sent by the first Intel rep after the chat was ended asking if they can close the ticket, again, reiterating that I should not have to RMA my processor. Although they did add that the second rep did advise me to monitor my processor after the update and if it has further issues contact support.

 

Most people outside of Intel are skeptical and have been pretty sure that degraded processors will need to be RMA'd. I myself am included in thinking that an overvolted processor that has degraded and is unable to function at stock settings would need to be RMA'd. I mean a software update can't fix physical damage caused by the chip being overvolted.

 

Curious if anyone else was in my shoes. What you would do? Return the Z790 and DDR5 ram and just hope for a quick update for my Z690 board, return them and go for a platform swap to AMD? Or just keep the Z790 and DDR5 ram? I am aware that I most likely jumped the gun with purchasing the new board and ram, I've just been kind of frustrated with the whole situation and having to deal with it since my processor initially started having the issue.

IMO RMA the CPU after the micro-code update, it's damaged goods. Get the new CPU. Then decide what to do. You have some time to deal with this. Otherwise I think waiting for Ryzen 9000 on AM5 makes a lot more sense than buying 7000 right now.

[leadeater]

6 hours ago, SpookyCitrus said:

-snip-

FYI You can update your CPU microcode without a BIOS update. The easiest way is actually by installing Linux then following the microcode update documentation which is essentially acquiring the latest microcode file and putting it in the correct directory then reboot.

 

The update is permanent so your Linux install is literally temporary just to update the CPU microcode, after you're done you can go back to Windows so if you have a spare SSD or even HDD laying around just disconnect or disable your current OS storage device and use that spare one for the Linux install.

 

https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/secure-coding/loading-microcode-os.html

https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files

[Lightwreather]

42 minutes ago, leadeater said:

The update is permanent so your Linux install is literally temporary just to update the CPU microcode

Would live booting work off of an install drive?

[leadeater]

3 minutes ago, Lightwreather said:

Would live booting work off of an install drive?

 I'm not sure Live goes through the same boot process . You can check the microcode version so you can tell if it worked, you can put all the files etc in the Live media so I don't see how it would hurt to try.

[SpookyCitrus]

2 hours ago, leadeater said:

FYI You can update your CPU microcode without a BIOS update. The easiest way is actually by installing Linux then following the microcode update documentation which is essentially acquiring the latest microcode file and putting it in the correct directory then reboot.

 

The update is permanent so your Linux install is literally temporary just to update the CPU microcode, after you're done you can go back to Windows so if you have a spare SSD or even HDD laying around just disconnect or disable your current OS storage device and use that spare one for the Linux install.

 

https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/secure-coding/loading-microcode-os.html

https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files

Great to know. I had no idea that was a thing.

[porina]

3 hours ago, leadeater said:

The update is permanent

Are you sure about that? My understanding was that it is volatile. The github link you provided states:

Quote

Microcode states are reset on a power reset, hence its required that the MCU be loaded every time during boot process.

In the distant past non-K overclocking of Skylake was patched out in newer microcode. Since the OS loaded update microcode at runtime, that OC would no longer work unless you intentionally used an older OS. In theory you could try to stop the OS from updating the microcode but I never tried that.

[leadeater]

35 minutes ago, porina said:

Are you sure about that? My understanding was that it is volatile. The github link you provided states:

Ah sod, you're right. That's annoying because I thought they were permanent like many/all of the BIOS microcode updates. I've had CPUs that have been microcode updated not work in the exact same motherboard model just older BIOS/firmware version because the CPU had been microcode updated. That's pretty old Xeon CPUs though so it may not work like that at all anymore.

 

I guess it is nice having it temporary though i.e. trying the early Side Channel updates and deciding too much performance loss.

 

@SpookyCitrus Flag it, the main Intel documentation link didn't say it was temporary but the github does which I didn't read since I read the other one.

 

That said Windows also does this too and you can get updated microcode installed through Windows Update so you may end up getting covered that way since it's significant enough to get put through that distribution channel.

[Bitter]

4 hours ago, leadeater said:

 I'm not sure Live goes through the same boot process . You can check the microcode version so you can tell if it worked, you can put all the files etc in the Live media so I don't see how it would hurt to try.

You shouldn't but you can install to a USB drive like it was a permanent system drive, you just need to make sure the system drives are 'turned off' in BIOS. If it's a USB drive with at least 200MB/S burst write it's tolerable to use for a short period of time. I had an install on a 128GB thumb drive that would write at 350 for about 4GB before dropping down slower. It was a usable experience but install and large updates too a while. A possible alternative for those without a spare SSD or you can get an external SSD adapter and cheap SSD from Amazon.

[leadeater]

4 hours ago, Bitter said:

You shouldn't but you can install to a USB drive like it was a permanent system drive, you just need to make sure the system drives are 'turned off' in BIOS. If it's a USB drive with at least 200MB/S burst write it's tolerable to use for a short period of time. I had an install on a 128GB thumb drive that would write at 350 for about 4GB before dropping down slower. It was a usable experience but install and large updates too a while. A possible alternative for those without a spare SSD or you can get an external SSD adapter and cheap SSD from Amazon.

See the 2 posts above, seems like it's a pointless suggestion anyway 

[Bitter]

47 minutes ago, leadeater said:

See the 2 posts above, seems like it's a pointless suggestion anyway 

I also was under the impression that firmware and microcode applied in Linux was permanent through to other OS's.

Would suck if someday you do a clean install, go game, and cook your 14th gen CPU.

[starsmine]

15 hours ago, Poinkachu said:

 

I think it's quite funny that the Oxidation problem confirmation is just at Reddit :x

I dont think the oxidation issue is particularly wide spread, nor would it be an issue other then it last 7 years rather then 10 if the microcode wasnt pushing incorrect voltages. 

I honestly think Steve is going way too hard on that point, when he has not confirmed it himself yet to be an issue or the scale of it. 

[Kisai]

Keep in mind there are a bunch of other features that can cause instability problems. Like enabling ASPM on the ASUS boards will cause the GPU to lockup and the windows log full of PCIe devices failing.

 

On the stock firmware, the 14th gen CPU wouldn't even boot with XMP turned on. The BIOS update had to be done with a lot of defaults forced to slower settings. After that, every BIOS update since May has been fine as long as ASPM was not enabled.

 

The stability issues from TVB I probably didn't run into at all, most likely because I disable all the "AI overclocking" stuff as a matter of wanting the device to be stable.

 

[05032-Mendicant-Bias]

I got my 13700 in early 2023, so potentially could be an affected part.

 

My system has been rock solid. I can't tell if it is affected or not...

 

Unfortunately I don't have a pic of the IHS, I made a video of the build but I can't read the fine prints. I would have to remove the CPU to read the IHS markings. I'll update the MOBO when the new BIOS come out and keep going as is.