Intel says their 13th/14th Gen instabilities are from 'elevated operating voltages stemming from a microcode algorithm'

[Kisai]

59 minutes ago, porina said:

From your earlier post the error showed at 10240k FFT, which would normally point to a IMC/ram problem if it doesn't also happen at much smaller FFTs.

It went through Iteration 1 with no fails while XMP was not turned on. (Which means the CPU was also operating at a lower voltage) However chrome tabs are still randomly crashing.

[05032-Mendicant-Bias]

11 hours ago, Kisai said:

It went through Iteration 1 with no fails while XMP was not turned on. (Which means the CPU was also operating at a lower voltage) However chrome tabs are still randomly crashing.

Try turning off the browser hardware acceleration, in the past this solved an instability with my old AMD GPU.

[Kisai]

1 hour ago, 05032-Mendicant-Bias said:

Try turning off the browser hardware acceleration, in the past this solved an instability with my old AMD GPU.

The thing is, everything was working fine until a few days ago, and only two things changed in that time:

GPU driver updated (which I rolled back)

Windows Update (2024-08 Cumulative Update Preview for Windows 11 Version 23H2 for x64-based Systems (KB5041587))

 

Now it IS possible that both Firefox and Chromium also updated silently at the same time, but what are the odds that they broke the same thing?

[Bitter]

Didn't that Windows patch also have some 'performance enhancements'? Can you roll back to a restore point prior to that patch installing?

[Kisai]

41 minutes ago, Bitter said:

Didn't that Windows patch also have some 'performance enhancements'? Can you roll back to a restore point prior to that patch installing?

Pretty sure the problem is hardware, the trouble is determining if it's the CPU, it's memory controller, the voltages, or the RAM itself.

 

Memtest86 with all 4 modules in XMP 5600 mode had errors (3rd digit from the right would show an error) on test 4 5 or 6, so I pulled the second and fourth module (this was a 128GB DDR5 kit, so this takes it down to 64GB) and it passed. What is also interesting is that after I pulled them, the PEG boot started working again (it would only show the BIOS on the IGPU the last few days.)

 

So unless that windows patch is pushing a microcode change, none of this makes any sense.

[StDragon]

2 hours ago, Kisai said:

So unless that windows patch is pushing a microcode change, none of this makes any sense.

If it's failing on a specific address and not random, that indicates a faulty chip on the DIMM. It happens.

[Kisai]

1 minute ago, StDragon said:

If it's failing on a specific address and not random, that indicates a faulty chip on the DIMM. It happens.

The modules I pulled were A1 and B1, which to me suggests that the error had to be in A1, but the addresses listed suggested it was in A2 and B1. Because I can't remove the modules without removing one of the CPU fans from the tower cooler, I'm a but hesitant to fidgit with this too much in case the problem magically fixes itself once the non-beta bios to fix the the intel CPU issues is released. I have an entire story about experiencing this problem with on the DDR2 system and I wound up downgrading the CPU on that one.

 

[Dabombinable]

18 hours ago, Kisai said:

The modules I pulled were A1 and B1, which to me suggests that the error had to be in A1, but the addresses listed suggested it was in A2 and B1. Because I can't remove the modules without removing one of the CPU fans from the tower cooler, I'm a but hesitant to fidgit with this too much in case the problem magically fixes itself once the non-beta bios to fix the the intel CPU issues is released. I have an entire story about experiencing this problem with on the DDR2 system and I wound up downgrading the CPU on that one.

 

I'd make note of which module was where originally, and then start swapping them around. It working with 3 sticks of RAM, and PEG boot functioning once more could be a symptom of the IMC having problems. And it would be under less stress.

The P45 NB on my old Asus P5Q Deluxe did actually behave in a similar manner as its memory controller degraded (went from doing 1600MHz FSB+4x DDR2 1066 all day to 1600MHz FSB and single channel DDR2 800 stable).

[05032-Mendicant-Bias]

On 7/27/2024 at 11:27 AM, 05032-Mendicant-Bias said:

I have a 13700F bought in early 2023, so within the affected range.

 

My system is rock solid, by far the best computer I built. I don't remember when the last hard crash was. Viktoria 3 and Cyberpunk sometime do crash to desktop, but I doubt it's the CPU fault there.

...

I run Cinebench 2024 before and after the BIOS update and the results are painful... multicore I went from 1561 to 1066, singlecore sayed the same at 113.

 

I got around to updating from 11b to 11d on my Gigabyte B760 Gaming X. I applied XMP, the intel detection tool reports 0x129 and passes all tests.

 

I redid the benchmarks, and I'm happy to report that I have regained almost all the lost performance! 1066 to 1524 Now it boosts to 4.6Ghz all cores.

[porina]

https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239

Another update from Intel who have further narrowed down the part of the CPU that is getting hit by the voltages is a clock tree, leading to a duty cycle shift and the instabilities.

 

They will be releasing another microcode to resolve another contributor state of elevated voltages while the CPU is idle or under light loads. This microcode will be 0x12B and performance of it is claimed to be within measurement tolerance of 0x125. I've not kept track of which version did what to performance but it was interesting it wasn't compared to previous release 0x129.

[Mark Kaine]

On 9/7/2024 at 9:42 AM, 05032-Mendicant-Bias said:

 

I got around to updating from 11b to 11d on my Gigabyte B760 Gaming X. I applied XMP, the intel detection tool reports 0x129 and passes all tests.

 

I redid the benchmarks, and I'm happy to report that I have regained almost all the lost performance! 1066 to 1524 Now it boosts to 4.6Ghz all cores.

ya cool... so what did intel actually do to prevent the *hardware* faults that are occurring? 

 

or in other words, how long do you expect the chip to last?

 

 

(on other hand you probably can just keep rma'ing it indefinitely lol)

 

 

37 minutes ago, porina said:

https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239

Another update from Intel who have further narrowed down the part of the CPU that is getting hit by the voltages is a clock tree, leading to a duty cycle shift and the instabilities.

 

They will be releasing another microcode to resolve another contributor state of elevated voltages while the CPU is idle or under light loads. This microcode will be 0x12B and performance of it is claimed to be within measurement tolerance of 0x125. I've not kept track of which version did what to performance but it was interesting it wasn't compared to previous release 0x129.

that's the thing they keep releasing "microcodes" but I've yet to see evidence this actually improves longevity... only time will tell ig, however there are already reports of "updated" microcodes crashing just like before.  

[porina]

42 minutes ago, Mark Kaine said:

that's the thing they keep releasing "microcodes" but I've yet to see evidence this actually improves longevity... only time will tell ig, however there are already reports of "updated" microcodes crashing just like before.  

How do you propose it be proven? Run 100 systems for a year or two? See you in 2026. If the cause is known, removing the cause should fix it. There may be some risk there remains some other unknown mechanism in play, but given the focus this has had it gets ever less likely.

 

As for crashes, that could be from anything. My AMD system has had crashes. Must be Intel microcode causing it. If you want to blame the microcode, the burden of proof is to show if those crashes are due specifically to that and not other reasons. Even if they are proven not to be stable, warranty exists.

[Kisai]

1 hour ago, porina said:

https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239

Another update from Intel who have further narrowed down the part of the CPU that is getting hit by the voltages is a clock tree, leading to a duty cycle shift and the instabilities.

 

They will be releasing another microcode to resolve another contributor state of elevated voltages while the CPU is idle or under light loads. This microcode will be 0x12B and performance of it is claimed to be within measurement tolerance of 0x125. I've not kept track of which version did what to performance but it was interesting it wasn't compared to previous release 0x129.

I actually did the RMA request for the CPU yesterday because the CPU started doing the random crashes and I couldn't into Windows at any stable RAM settings until I reset the factory settings again and touched NOTHING. So it's not even running XMP right now.

[StDragon]

5 hours ago, porina said:

Another update from Intel who have further narrowed down the part of the CPU that is getting hit by the voltages is a clock tree, leading to a duty cycle shift and the instabilities.

If Arrow Lake is still undergoing engineering, hopefully they'll ensure this clock tree circuitry is designed to be more robust. Otherwise, they'll have to wait until the next stepping. 

[porina]

23 minutes ago, StDragon said:

If Arrow Lake is still undergoing engineering, hopefully they'll ensure this clock tree circuitry is designed to be more robust. Otherwise, they'll have to wait until the next stepping. 

Other way around. Fix the cause, not the symptom. The clock tree degraded first due to the elevated voltages. The fix is to not have the elevated voltages at all, which is what the microcode is doing.

 

While not official, the expected release of Arrow Lake could be within a month. Not saying they need to, but they're not changing silicon before then. Product going on sale would already have been made. If they did need any short term adjustments, that could be delivered through microcode before release. Ongoing improvement to product is a normal part of operation so there may be refinements over time regardless.

[05032-Mendicant-Bias]

On 9/26/2024 at 11:23 AM, Mark Kaine said:

or in other words, how long do you expect the chip to last?

No clue. My system has been always flawless with no crashes, I have no complaints there. I trust Intel enough to do an RMA if a problem does arise.  I'm having to do more maintenance than I'd like, I pick Intel because I trust it to be more stable than AMD, so I'm not happy about the poor maintenance that my CPU is requiring. My expectation was the same as my 8700, that it would outlast the system and the system after that, and the system after that.

'll do another bios update and benchmark the result to document the performance.

My complaint is that Intel kept the problem under the rugs for one year, letting the electromigration issue festers. I'm actually happy that Intel is right now staying atop of the issue and monitoring the situation. it's what they should have been doing from the start.

 

This is feeling like the Beoing event for Intel. Prioritizing dividends and stock buybacks and short term shareholder value over making good products for a decade is finally hurting Intel where it stings, the stock price. Now I'm confident the problem will finally get fixed and culture improves.

I hate public companies that rewards CEO with stocks options. It prioritizes making short term moves, and hurt the businness in the long run, which wipes out shareholders anyway..

[Mark Kaine]

30 minutes ago, 05032-Mendicant-Bias said:

My complaint is that Intel kept the problem under the rugs for one year, letting the electromigration issue festers. I'm actually happy that Intel is right now staying atop of the issue and monitoring the situation. it's what they should have been doing from the start.

tbf i just read not all chips have this issue... which still hints to a manufacturing defect imo, but at least if you're lucky it will never exhibit these issues (if true) and yes the way this has been handled so far is the worst part - even though i understand why they did this, it just makes financially the most sense... it's a catastrophic situation anyhow, so why not do "damage control" wherever possible. 

 

And yes, RMA is obviously an option... personally for me this is just too inconvenient, intel has these issues for a while now, either too high voltages at default,  crazy boost algorithms, no multi-threading or dodgy "branch prediction" to name a few lol...  Ryzen has also issues, mainly with memory controllers but that seems tame in comparison. 

Additionally intel is usually more expensive and i hate the retention mechanism they use (also why i won't buy current gen AMD lol, i just hate it )

 

ps: and yep it's Boeing all over again,  who also seemingly can't catch a break - it takes very long to fix these fundamental issues that apparently started decades ago, and its unclear if its even possible to be fixed... 

[porina]

42 minutes ago, 05032-Mendicant-Bias said:

My complaint is that Intel kept the problem under the rugs for one year, letting the electromigration issue festers. I'm actually happy that Intel is right now staying atop of the issue and monitoring the situation. it's what they should have been doing from the start.

Do we have a timeline for this? It feels like a long time since initial reports that something might be happening, but given the nature of the problem it isn't the sort of thing you can diagnose quickly. It also took some time before it built up enough momentum to get attention.

 

42 minutes ago, 05032-Mendicant-Bias said:

I hate public companies that rewards CEO with stocks options. It prioritizes making short term moves, and hurt the businness in the long run, which wipes out shareholders anyway..

Shares are probably better than the alternative of cash, since shares will change value with company performance and at C level, there is more friction on selling them than for the little people.

[Zodiark1593]

On 9/5/2024 at 6:26 AM, Kisai said:

Pretty sure the problem is hardware, the trouble is determining if it's the CPU, it's memory controller, the voltages, or the RAM itself.

 

Memtest86 with all 4 modules in XMP 5600 mode had errors (3rd digit from the right would show an error) on test 4 5 or 6, so I pulled the second and fourth module (this was a 128GB DDR5 kit, so this takes it down to 64GB) and it passed. What is also interesting is that after I pulled them, the PEG boot started working again (it would only show the BIOS on the IGPU the last few days.)

 

So unless that windows patch is pushing a microcode change, none of this makes any sense.

Memory issues are an absolute bastard to troubleshoot…

 

Had issues with my sister’s PC (Ryzen 5 5600G using iGPU), but they only cropped up under gaming loads, and presented as GPU driver crashing. Memory was the first thing I suspected, but Memtest passed every time. 
 

Removing what ended up being the faulty stick resolved the GPU driver crashes (with an RMA following), but damn that gaslighting. Learned to not trust MemTest. 

[Bitter]

Memtest is good for testing the memory but not necessarily finding out if the memory is stable enough to work all the time. I've ran into that too. Prime95 blend is pretty good for sussing out memory issues.

[StDragon]

33 minutes ago, Bitter said:

Prime95 blend is pretty good for sussing out memory issues.

What you really want is to test all of the CPU registers and ISA under load, not just a tiny subset such as Prime95 or Quick Sync codec encoding which only uses a specific subset.

 

Gaming is pretty good at utilizing most of the CPU, but a specific CPU diagnostic tool is preferred. The Intel Processor Diagnostic Tool (IPDT) namely.

 

https://www.intel.com/content/www/us/en/download/15951/intel-processor-diagnostic-tool.html

 

if only AMD has the equivalent of IPDT

[Bitter]

4 minutes ago, StDragon said:

What you really want is to test all of the CPU registers and ISA under load, not just a tiny subset such as Prime95 or Quick Sync codec encoding which only uses a specific subset.

 

Gaming is pretty good at utilizing most of the CPU registers, but a specific CPU diagnostic tool is preferred. The Intel Processor Diagnostic Tool (IPDT) namely.

 

https://www.intel.com/content/www/us/en/download/15951/intel-processor-diagnostic-tool.html

 

if only AMD has the equivalent of IPDT

True. It can take blend a while to fail but it's been pretty reliable at finding bad memory over the past 20 years for me.

[porina]

20 minutes ago, StDragon said:

What you really want is to test all of the CPU registers and ISA under load, not just a tiny subset such as Prime95 or Quick Sync codec encoding which only uses a specific subset.

I think the earlier comment was focused more on ram related problems, not general stability. In past manual overclocking of ram I've also encountered a case where memtest would run as long as you like without error, but Prime95 blend would fail in seconds. I knew the CPU was good so it was the ram settings that was the problem.

 

Like any stability testing, it can help to run several tools. Now whenever I have suspect ram, I'd run a mix of memtest, P95 blend, and Aida64 stability test with only ram selected. For P95 and Aida, I find a few minutes can give a quick sanity check although longer can always be used for more confidence.

 

For Prime95 specifically, the memory access pattern switches a lot between reads and writes. I'm not sure if that particular pattern is exercised by memtest where it might focus on one thing or other, and I have no idea what aida does.

[StDragon]

25 minutes ago, porina said:

For Prime95 specifically, the memory access pattern switches a lot between reads and writes. I'm not sure if that particular pattern is exercised by memtest where it might focus on one thing or other, and I have no idea what aida does.

The list of tests are here.

The closest tests to mimic Prime95 would be Test# 6 (BurnBX). Though #8, #11, #12, and #13 should be catching errors too.

Hypothetically, Prime95 pushes past the breaking point due to excessive thermal load beyond what Memtest does. I can only surmise that if the memory controller isn't able to function properly with the DIMMs, it's the extra degrees in heat generated that pushes the limit; but that's conjecture on my part.
 

[porina]

24 minutes ago, StDragon said:

The closest tests to mimic Prime95 would be Test# 6 (BurnBX). Though #8, #11, #12, and #13 should be catching errors too.

Note sure any of those are close to what Prime95 access pattern might be like, but there's not enough detail to be sure. #6 is a move, which while it might involve reads and writes, might differ from actual separate reads and writes. #8+11-13 sounds like they might operate over large areas.

 

In Prime95, each worker has an associated data size, which is 8x the FFT size. If the total size required from all workers exceeds the CPU's cache, it spills over to ram. Each worker runs independently so accesses can get interleaved between them.

 

24 minutes ago, StDragon said:

Hypothetically, Prime95 pushes past the breaking point due to excessive thermal load beyond what Memtest does. I can only surmise that if the memory controller isn't able to function properly with the DIMMs, it's the extra degrees in heat generated that pushes the limit; but that's conjecture on my part.

Higher thermal load, yes. Breaking point? Only if you have a badly configured system. Higher thermal load impacting the IMC could be a factor but it shouldn't be anything a properly stable system can't handle. Note this is more of a potential risk area for 14nm era Intel CPUs. Since then the loss of AVX-512 combined with the newer process node used lowers the relative loading of 12+ gen. On recent AMD it is a totally different story where AVX-512 is more efficient than AVX2, but that's really getting off topic.