The MBP throttling issue isn't what everybody thinks...

[Master Disaster]

So it turns out this issue isn't as simple as everybody thinks, a random person (that's actually his username) on Reddit did some digging and found out exactly what's going on.

 

The issue has nothing to do with the CPU cooling solution and nothing to do with power profiles, turns out Apple didn't fit VRMs on the board capable of powering the I9 CPU. Ok so that's not 100% accurate, the VRMs would be fine if they had adequate airflow to cool them but as they're sealed into an enclosed space with no active cooling they're getting very toasty and the CPU is throttling to save the VRMs from burning out.

Quote

Hello all,

 

tl;dr ... I found the magical command which makes my i9 MBP capable of Cinebench scores of ~1100 on the first run, and 1000-1100 on each successive run, without throttling even if I run it back-to-back-to-back-to-back.

 

Like many here, I've got an i9 2018 MBP, and I've also experienced the throttling as others have seen.

 

As you may have read on other threads, the ultimate root cause of the very bad performance drop during the throttling is not thermal throttling of the CPU, but rather power throttling of the VRM (voltage regulator module), being unable to satisfy the power desires of the i9 CPU.

 

When the VRM maxes out (overheats ... but this is different from CPU thermal throttling), the motherboard sends a signal to the CPU to drop it's speed to minimum (800Mhz) to allow the VRM a chance to cool down. The CPU then returns to it's previous desire to pull maximum power, spins up to high turbo speeds, and the cycle repeats again. When the CPU keeps switching from Turbo to 800Mhz, it is in a very inefficient state, so the amount of work being done relative to the amount of power drawn decreases.

He has a "fix" that he claims can improve performance by upto 20% but we warned, it involves disabling SIP and editing system voltages manually!!

Spoiler

Quote

Warning: To do this, you will have to put your system in a "custom" SIP (System Integrity Protection) state to allow unsigned KEXT (Kernel Extensions), this reduces security on your system somewhat, but this is at least not completely disabling SIP.

Warning2: If you do this, and it damages your laptop, and Apple does not cover your warranty, that is your own problem. You are doing this at your own risk. With that said, In my personal opinion, the default state is actually more damaging to the laptop than this custom state (As you will see below).

 

1) install https://github.com/sicreative/VoltageShift

 

2) CHANGE DIRECTORY to where voltageshift was installed (you can't just run it with a full path in the command line, there is a bug in it which makes it require you actually being in the directory and writing ./voltageshift ... not ~/voltageshift/voltageshift)

 

3) sudo chown -R root:wheel VoltageShift.kext

 

4) ./voltageshift read 0x610

 

RDMSR 610 returns value 0x4283e800dd8320

 

I'll translate what the important bits here means: MSR 610 is the MSR_TURBO_POWER_LIMIT Hex 3e8 == 1000 1000 / 8 == 125 This means that PL2 (Power_Limit_2) is set to 125 Watts. Power Limit 2 is the "Turbo Boost" limit.

 

Hex 320 == 800 800/8 = 100 This means that PL1 (Power_Limit_1) is set to 100 Watts. Power Limit 1 is the "Steady State" limit, ie: how much power the CPU is configured to try and pull for extended periods of time.

 

The math for the timing variable is complicated, so just take my word for it, but: Power_Limit_1_Time ... dd = 28 Seconds.

 

So, Apple has shipped the i9 Macbook Pro with it's CPU power regulation set to to 125 Watt Turbo for 28 seconds, 100 Watt Steady state.

 

I've spent a few hours testing at many different draw rates until I could find the rate at which the VRM does not max out, for my machine, that is 49 Watts. Note that this does not include power draw for other accessories (ie: the GPU), so if you are drawing heavy power using other chips, you might need to set the limit lower than 49 watts.

 

I've also spent a few hours testing the maximum amount of time you can draw in Turbo from the VRM without maxing it out, and for my machine, it is 8 seconds.

 

So, putting this all together ... 49 * 8 = 392 392 in hex is 188

 

Like I said earlier, the time values are complicated, so just take my word for it, but ...

D9 = 7 seconds.

1B = 8 seconds.

9B = 10 seconds.

[...]

DD = 28 seconds. (default)

Compiling this all together leads to the CORRECT MSR for this machine: 0x4283E8001B8188

 

5) And you can set this using this command: ./voltageshift write 0x610 0x4283E8001B8188

Note, this gets overwritten every time you reboot, so if you want to set this, you will need to do it on boot. I'll probably write up a launchd script to do this.

 

In my opinion, Apple seriously screwed up here. The testing that I did was easily within Apple's capabilities, and it is highly irresponsible of them to ship these machines out this way. As far as I am aware, the MSR is set to the intel's default ... and clearly Intel expects that if you are going to use this default, you will have a machine capable of delivering 100 Watts of power steady state.

 

Intel does not expect that you must have thermal capabilities of shedding those 100 Watts, as the chip will thermally throttle if it reaches TJunction (100C), and that thermal throttling would not be nearly as painful as this type of VRM throttling. Most likely, if Apple wanted to handle things that way, they could have figured out a way to attach the heatpipe to the VRM. Even if they did not want to handle things that way, they should have reduced the MSR value from the factory to something that the VRM is actually capable of delivering (ie: My settings).

 

[Edited to add note: From messages others have sent me, it seems that this MSR is actually not Intel's default, but rather this is what Apple has been setting the MSR value to on all intel chips in Macbook Pro and Macbook Air for a number of years ... This is just the first chip to actually be capable of drawing enough power for this to matter. Previous chips just couldn't draw that much power, so they never hit the VRM limit before]

 

The good news is that these settings increase the performance by at least 20% (and possibly much more depending on the workload), and Apple can adjust this setting (or handle the VRM overload using other power management logic). Even if Apple doesn't, you as a consumer can do this yourself.

 

The bad news is that it's clear that this i9 chip can probably do signifigantly better if it were driven with a better VRM (and had adequate cooling as well), and could probably get 20-25% faster speeds than with my settings if the power system were up to task.

 

There is no way that Apple can get that additional gain with the current hardware design, short of a recall to change out the heatsink to something that cooled the VRMs, and replace the thermal paste with something like Liquid Metal. It's pretty clear that there is no additional space for bigger fans, so apple's options here are very limited.

 

The even worse news: If apple does nothing, and you do not change the MSR settings, when your system throttles this way and alternates back and forth between 800Mhz and full-Turbo, it is putting an EXTREME load on the VRM ... This will almost certainly shorten the lifespan of the VRM causing early logic board failure.

 

If anyone has a 2018 15" MBP with the i7 2.6 chip, I'm very interested to hear what the default MSR settings are for that chip, and also how Cinebench scores look with my tuned settings. These tuned settings should be acceptable for any 2018 15" MBP.

 

Of course, as I said above, if you have a heavy GPU load, this does not leave any budget for that, so you will need to adjust your settings accordingly (and that's homework for you -- I don't use my GPU in heavy sustained workloads ever). Also, these settings assume reasonably sane cooling environment ... Ie: sitting on a desk in an air conditioned room. If you are testing this outdoors at 100F, I'd assume you will still get VRM throttling. If you are testing in a freezer, or outdoors in the wintertime ... you might be able to get more than 49W steady-state.

 

https://www.reddit.com/r/macbookpro/comments/91256u/optimal_cpu_tuning_settings_for_i9_mbp_to_stop/

 

 

So there you have it folks, the issue is hot VRMS not hot CPU.

[orbitalbuzzsaw]

Neat investigation but same result - the CPU throttles and performance gets shot to hell

[Master Disaster]

Just now, orbitalbuzzsaw said:

Neat investigation but same result - the CPU throttles and performance gets shot to hell

Yeah, as Louis points out it turns out it's worse than everybody thought, the user can't crack open and liquid metal the cooler because it doesn't cover the VRMs.

[TheGlenlivet]

Maybe Apple never turned it on in the lab when they built it.....

/s

[orbitalbuzzsaw]

1 minute ago, Master Disaster said:

Yeah, as Louis points out it turns out it's worse than everybody thought, the user can't crack open and liquid metal the cooler because it doesn't cover the VRMs.

I mean true but how many people who'd buy a Mac would do that anyway?

[suicidalfranco]

now let's see the applelogist try to shift the blame to intel

[Master Disaster]

I'd love to see Linus/Anthony try the fix out on a livestream.

[dalekphalm]

My guess is that they didn't redesign the power delivery system that was used in the lower CPU models - and therefore was simply insufficient for the i9 power requirements.

 

Whether they tested this or knew it would happen? I suspect they did, and simply chose to ignore the results.

[TigerHawk]

I feel like the only people reading/investigating/promoting? this is the PC community. I doubt Apple users are even aware. They have their iBlinders on

[Master Disaster]

Just now, dalekphalm said:

My guess is that they didn't redesign the power delivery system that was used in the lower CPU models - and therefore was simply insufficient for the i9 power requirements.

 

Whether they tested this or knew it would happen? I suspect they did, and simply chose to ignore the results.

Yep, that was exactly my first thought too, they literally took an I7 machine, dropped an I9 in then shipped it. It's funny because the hackintosh community already did most of the hard work getting I9s running on macOS, all Apple had to do was get the hardware right.

[Hunter259]

Yeah no. It's definitely a bit a both. These machines under full load have the CPU near 100c the entire time. The 800MHz drops are explained by this for sure but the throttling is mostly due to insufficient cooling.

[orbitalbuzzsaw]

1 minute ago, Master Disaster said:

Yep, that was exactly my first thought too, they literally took an I7 machine, dropped an I9 in then shipped it. It's funny because the hackintosh community already did most of the hard work getting I9s running on macOS, all Apple had to do was get the hardware right.

Yeah well Apple doesn't give a shit about people who actually know something about hardware

[Master Disaster]

1 minute ago, Hunter259 said:

Yeah no. It's definitely a bit a both. These machines under full load have the CPU near 100c the entire time. The 800MHz drops are explained by this for sure but the throttling is mostly due to insufficient cooling.

Sorry buddy but the proof is in the fix, by reducing VRM voltage only he was able to sustain a 20% increase in Cinebench performance, repeatable multiple times with no throttling.

[Hunter259]

Just now, Master Disaster said:

Sorry buddy but the proof is in the fix, by reducing VRM voltage only he was able to sustain a 20% increase in Cinebench performance, repeatable multiple times with no throttling.

His chip is still under the 2.9GHz base clock...

[seon123]

16 minutes ago, orbitalbuzzsaw said:

@DrMacintosh

He came out on the previous thread going against Apple's decision...

He likes Apple, but he's not completely unreasonable, unlike some others

Edited July 23, 2018 by seon123
Something something

[Master Disaster]

5 minutes ago, Hunter259 said:

His chip is still under the 2.9GHz base clock...

Where you getting that from, his chip is running at 3.2Ghz sustained and touching 4Ghz at times. Look at the screenshot in the Reddit post.

[Tabs]

16 minutes ago, TigerHawk said:

I feel like the only people reading/investigating/promoting? this is the PC community. I doubt Apple users are even aware. They have their iBlinders on

I don't know about that. There are a lot of people who buy Apple products because they are part of the ecosystem or don't know any better or simply choose them over alternatives - but the overlap between people who rush out and drop $3000 on a laptop within days of launch and people who fit in the previous category is pretty damn slim.

 

Most of the early reports of this problem came from people who bought these devices specifically to review what should be one of the fastest portable machines in the world - and who traditionally give glowing reviews of them. That's when you know Apple has a real problem on their hands here.

 

The people with their "iBlinders" on are never going to drop 3 grand on a new mac, and the professionals who need the potential performance of the i9 in this form factor are all going to be INCREDIBLY unhappy about this.

 

Apple has been lazily sticking "new" hardware in the same shell now for 3 years, without considering the power requirements of having more cores and a higher performance part. Everyone expected this the moment they announced the specs, but even the diehard apple fans haven't been able to justify it.

 

Edit: I forgot to add, I don't think this community should stoop to lobbing vitriol at people like @DrMacintosh (not so much in this thread, but I've seen it frequently over the past few days). He works in the industry of repairing apple devices, and he knows the score. He's not some blind idiot who buys whatever crap Apple drops only because he likes the company.

Edited July 23, 2018 by Tabs

[Hunter259]

Just now, Master Disaster said:

Where you getting that from, his chip is running at 3.2Ghz sustained and touching 4Ghz at times. Look at the screenshot in the Reddit post.

Yes. Look at Prime. 2.8GHz with 0 fluctuation. It's great it doing the quick runs at higher than base now. That's fantastic. But under more severe loads, and especially combined CPU+GPU loads, the chip WILL throttle due to insufficient cooling.

[Dabombinable]

The CPU still hits 100^oC.....so how is the cooling not at fault?

[Master Disaster]

2 minutes ago, Hunter259 said:

Yes. Look at Prime. 2.8GHz with 0 fluctuation. It's great it doing the quick runs at higher than base now. That's fantastic. But under more severe loads, and especially combined CPU+GPU loads, the chip WILL throttle due to insufficient cooling.

That's a torture test dude, small FFTs is designed to hit the CPU as hard as possible, ofc it's throttling back. It is still a thin and light.

 

In the much more real world Cinebench the CPU is behaving normally.

 

He does mention that GPU load will probably push it back over in the fix.

[Hunter259]

1 minute ago, Master Disaster said:

That's a torture test dude, small FFTs is designed to hit the CPU as hard as possible, ofc it's throttling back. It is still a thin and light.

 

In the much more real world Cinebench the CPU is behaving normally.

 

He does mention that GPU load will probably push it back over in the fix.

That was a large FFT test.

[Master Disaster]

5 minutes ago, Dabombinable said:

The CPU still hits 100^oC.....so how is the cooling not at fault?

It doesn't once you reduce VRM voltage, he did multiple Cinebench runs and it didn't go over 90c. (Ok one spike to 100)

 

https://www.imgur.com/a/M6mzJFV

[Master Disaster]

Just now, Hunter259 said:

That was a large FFT test.

Oh crap, my bad. I guess that's my punishment for scanning instead of reading. Actually that is an anomaly, large FFTs shouldn't cause that much stress.

[Hunter259]

2 minutes ago, Master Disaster said:

Oh crap, my bad. I guess that's my punishment for scanning instead of reading. Actually that is an anomaly, large FFTs shouldn't cause that much stress.

That's why it's able to sit that low in temps. Larges aren't nearly as demanding. The laptop still has hilariously poor cooling on top of nonexistent tweaked power limits.

[orbitalbuzzsaw]

19 minutes ago, seon123 said:

 not completely unreasonable

 

Okay, sure.