How to Undervolt NVidia GPUs in Linux

[Gorgon]

Linux users have been unable to under-volt NVidia GPUs since the Kepler days when NVidia removed that facility from the System Management Interface (SMI).

 

In essence, under-volting allows you to run a higher clock frequency at a lower GPU core voltage and thus at a lower heat load making GPUs more efficient.

 

The same effect can be had by lowering the power limit and adding a positive Clock Offset. The issue with this approach is when a Work Unit (WU) checkpoints, or something else occurs which reduces the GPU load for a longish interval, when returning to activity the offset tends to apply to an already fairly high clock frequency typically causing the WU to fail.

 

However, there is an often over-looked switch for the NVidia SMI that can be used to prevent this from happening.

    -lgc  --lock-gpu-clocks=    Specifies <minGpuClock,maxGpuClock> clocks as a
                                pair (e.g. 1500,1500) that defines the range
                                of desired locked GPU clock speed in MHz.
                                Setting this will supercede application clocks
                                and take effect regardless if an app is running.
                                Input can also be a singular desired clock value
                                (e.g. <GpuClockValue>).

So one could do something like:

nvidia-smi -pm 1
nvidia-smi -i 0 -pl 160
nvidia-smi -i 0 -lgc 1830,1980
DISPLAY=:0 XAUTHORITY=/run/user/121/gdm/Xauthority nvidia-settings \
 -a [gpu:0]/GPUFanControlState=1 \
 -a [fan:0]/GPUTargetFanSpeed=75 \
 -a [fan:1]/GPUTargetFanSpeed=75 \
 -a [gpu:0]/GPUPowerMizerMode=1 \
 -a [gpu:0]/GPUGraphicsClockOffsetAllPerformanceLevels=75

to:

1. Set Persistence Mode for all GPUs (so power limits "stick" across WUs)
2. Set the Power Limit for GPU0 to 160W
3. Set the Minimum GPU Core Clock to 1830MHz and the Maximum to 1980MHz.
4. Enable manual fan control
5. Set both fans for a Turing or later GPU to 75% (Pascal and earlier only have 1 fan control register which controls both fans)
6. Set GPU0 to "Prefer Maximum Performance"
7. Add a +75MHz (5 x 15MHz Turing "bins") GPU Clock offset

[svmlegacy]

Remember to enable coolbits, and could you explain to me how this actually undervolts the card? Have you tested the voltage behavior?

[Gorgon]

26 minutes ago, svmlegacy said:

Remember to enable coolbits, and could you explain to me how this actually undervolts the card? Have you tested the voltage behavior?

“... The same effect can be had ...”

 

I can confirm that using a positive clock offset at a reduced power limit will increase the effective performance (PPD) over a GPU with just the same power limit applied.

[specializedtom]

On 10/20/2020 at 8:54 AM, svmlegacy said:

Remember to enable coolbits, and could you explain to me how this actually undervolts the card? Have you tested the voltage behavior?

On the subject of undervolting and 'coolbits' : 

I have an RTX3070 and an RTX 2070 Super in my computer. I have no problems with enabling Coolbits to my RTX 3070 which is connected to a display. However I cannot enable Coolbits on my RTX 2070 Super that is not connected to a display. I need to be able to control the fan speed on my RTX 2070 Super so that it can keep it at a reasonable temperature during mining-intensive operations. My OS is Fedora 34 Server with Gnome 4.0 installed. 

If someone can point me in the right direction, it would be much appreciated. 

[svmlegacy]

5 hours ago, specializedtom said:

On the subject of undervolting and 'coolbits' : 

I have an RTX3070 and an RTX 2070 Super in my computer. I have no problems with enabling Coolbits to my RTX 3070 which is connected to a display. However I cannot enable Coolbits on my RTX 2070 Super that is not connected to a display. I need to be able to control the fan speed on my RTX 2070 Super so that it can keep it at a reasonable temperature during mining-intensive operations. My OS is Fedora 34 Server with Gnome 4.0 installed. 

If someone can point me in the right direction, it would be much appreciated. 

Start a separate thread on this issue instead of highjacking this one. 

[deama]

On 10/19/2020 at 11:31 PM, Gorgon said:

Linux users have been unable to under-volt NVidia GPUs since the Kepler days when NVidia removed that facility from the System Management Interface (SMI).

 

In essence, under-volting allows you to run a higher clock frequency at a lower GPU core voltage and thus at a lower heat load making GPUs more efficient.

 

The same effect can be had by lowering the power limit and adding a positive Clock Offset. The issue with this approach is when a Work Unit (WU) checkpoints, or something else occurs which reduces the GPU load for a longish interval, when returning to activity the offset tends to apply to an already fairly high clock frequency typically causing the WU to fail.

 

However, there is an often over-looked switch for the NVidia SMI that can be used to prevent this from happening.

    -lgc  --lock-gpu-clocks=    Specifies <minGpuClock,maxGpuClock> clocks as a
                                pair (e.g. 1500,1500) that defines the range
                                of desired locked GPU clock speed in MHz.
                                Setting this will supercede application clocks
                                and take effect regardless if an app is running.
                                Input can also be a singular desired clock value
                                (e.g. <GpuClockValue>).

So one could do something like:

nvidia-smi -pm 1
nvidia-smi -i 0 -pl 160
nvidia-smi -i 0 -lgc 1830,1980
DISPLAY=:0 XAUTHORITY=/run/user/121/gdm/Xauthority nvidia-settings \
 -a [gpu:0]/GPUFanControlState=1 \
 -a [fan:0]/GPUTargetFanSpeed=75 \
 -a [fan:1]/GPUTargetFanSpeed=75 \
 -a [gpu:0]/GPUPowerMizerMode=1 \
 -a [gpu:0]/GPUGraphicsClockOffsetAllPerformanceLevels=75

to:

1. Set Persistence Mode for all GPUs (so power limits "stick" across WUs)
2. Set the Power Limit for GPU0 to 160W
3. Set the Minimum GPU Core Clock to 1830MHz and the Maximum to 1980MHz.
4. Enable manual fan control
5. Set both fans for a Turing or later GPU to 75% (Pascal and earlier only have 1 fan control register which controls both fans)
6. Set GPU0 to "Prefer Maximum Performance"
7. Add a +75MHz (5 x 15MHz Turing "bins") GPU Clock offset

I'm a bit confused on why there should be a clock offset. Why not just run 

nvidia-smi -i 0 -lgc 1900,1900

and basically let it stay there? Wouldn't your clock now be at 1900 constantly? What's the point in the offset then? If it isn't stable at 1900 just set it to 1800?

[Gorgon]

2 hours ago, deama said:

I'm a bit confused on why there should be a clock offset. Why not just run 

nvidia-smi -i 0 -lgc 1900,1900

and basically let it stay there? Wouldn't your clock now be at 1900 constantly? What's the point in the offset then? If it isn't stable at 1900 just set it to 1800?

-lgc 1900,1900

will force the GPU to 1900MHz and will not allow it to clock down so when there is no load it will never go into the lower P-States and will consume much more electricity when idle. Setting a minimum clock of 0 allows it to go into it's lowest P-state.

 

With 0,1900 (or whatever) it will stay at the upper bound while doing work as long as it's not hitting a Power, Voltage or Temperature Limit.

 

For maximum efficiency while running distributed computing on Turing and Ampere GPUs I normally just run

nvidia-smi -pm 1
nvidia-smi -i 0 -lgc 0,<base_clock>

[deama]

5 hours ago, Gorgon said:

-lgc 1900,1900

will force the GPU to 1900MHz and will not allow it to clock down so when there is no load it will never go into the lower P-States and will consume much more electricity when idle. Setting a minimum clock of 0 allows it to go into it's lowest P-state.

 

With 0,1900 (or whatever) it will stay at the upper bound while doing work as long as it's not hitting a Power, Voltage or Temperature Limit.

 

For maximum efficiency while running distributed computing on Turing and Ampere GPUs I normally just run

nvidia-smi -pm 1
nvidia-smi -i 0 -lgc 0,<base_clock>

That makes sense.
Another question I had was how far down can you reduce the wattage? Like, say I had a 3090, and I wanted it to run at a max of 200W, is that possible with nvidia-smi? Run it at 200W and lock the base clock to a specific mhz?

[Gorgon]

On 10/27/2022 at 1:06 AM, deama said:

That makes sense.
Another question I had was how far down can you reduce the wattage? Like, say I had a 3090, and I wanted it to run at a max of 200W, is that possible with nvidia-smi? Run it at 200W and lock the base clock to a specific mhz?

How low the power can go depends on the BIOS version on the specific GPU.

 

If you look at the latest BIOS for the FE 3090, for example, has a Power Target of 350W and a Maximun of 400W with an adjustment range of -71% to +14% or 101.5 to 399W. Not that you'd want to run it at 100W as it would probably clock down well below the peak in the power efficiency curve.

 

Under Linux or Windows you can run nvidia-smi with a unreasonable power limit

nvidia-smi -i 0 -pl 1000

and it will throw an error telling you the Minimum and Maximum powers for the model. To see the Power Limits under Windows:

nvidia-smi -i 0 -q | find "Power Limit"

Linux:

nvidia-smi -i 0 -q | grep 'Power Limit'

Say you want to run your 3090 at a maximum clock of 1500MHz consuming no more than 200W you would run:

nvidia-smi -i 0 -pl 200
nvidia-smi -i 0 -lgc 0,1500

[deama]

10 hours ago, Gorgon said:

How low the power can go depends on the BIOS version on the specific GPU.

 

If you look at the latest BIOS for the FE 3090, for example, has a Power Target of 350W and a Maximun of 400W with an adjustment range of -71% to +14% or 101.5 to 399W. Not that you'd want to run it at 100W as it would probably clock down well below the peak in the power efficiency curve.

 

Under Linux or Windows you can run nvidia-smi with a unreasonable power limit

nvidia-smi -i 0 -pl 1000

and it will throw an error telling you the Minimum and Maximum powers for the model. To see the Power Limits under Windows:

nvidia-smi -i 0 -q | find "Power Limit"

Linux:

nvidia-smi -i 0 -q | grep 'Power Limit'

Say you want to run your 3090 at a maximum clock of 1500MHz consuming no more than 200W you would run:

nvidia-smi -i 0 -pl 200
nvidia-smi -i 0 -lgc 0,1500

Ah thanks.
I'm planning on building a new computer and going full linux at around Feb time, and wanted to get a 4090 rtx, but the ridiculous power of it is too crazy, so I'd like to cap it to 200W, even if the clock goes down a lot. Even if I get the 4080 or 4070 instead, it's not going to have as high of VRAM, as much cuda cores, and probably worse die so the clocks won't be as good at 200W as is for the 4090.

 

Do you happen to know if nvidia-smi works for newest cards or you have to wait for them to be added to the "repository" so to speak?

[Gorgon]

8 hours ago, deama said:

... Do you happen to know if nvidia-smi works for newest cards or you have to wait for them to be added to the "repository" so to speak?

It will work for cards at their launch

[deama]

Hey @Gorgon, sorry to bother you again, but I've just moved on over to linux with my nvidia card, and I just realised, that nvidia-smi is for x11 isn't it?
Is there a way to do it via wayland?

Specifically I'm on fedora gnome wayland.

[Gorgon]

7 hours ago, deama said:

Hey @Gorgon, sorry to bother you again, but I've just moved on over to linux with my nvidia card, and I just realised, that nvidia-smi is for x11 isn't it?
Is there a way to do it via wayland?

Specifically I'm on fedora gnome wayland.

Short answer. For now to enable over-clocking and fan control you'll have to switch from Wayland back to X11. Apparently installing the Nvidia binary (blob) drivers in Fedora does this for you (removes nouveau?)

 

nvidia-smi is a CLI tool and should just work under Wayland same as it does under X11 so you'd be able to apply a power-limit and/or lock the graphics clock but not apply an overclock or adjust the fan speeds.

 

Wayland is a PITA so I've just been disabling it on Ubuntu and using stock X11.

 

It appears that there is still no support in the Nvidia driver for overclocking and fan control under Wayland. This appears to be an issue with NVidia use of user-space controls as opposed to the kernel for over-clocking.

[K3Rn3L_P4n1c]

On 6/10/2021 at 2:26 AM, specializedtom said:

On the subject of undervolting and 'coolbits' : 

I have an RTX3070 and an RTX 2070 Super in my computer. I have no problems with enabling Coolbits to my RTX 3070 which is connected to a display. However I cannot enable Coolbits on my RTX 2070 Super that is not connected to a display. I need to be able to control the fan speed on my RTX 2070 Super so that it can keep it at a reasonable temperature during mining-intensive operations. My OS is Fedora 34 Server with Gnome 4.0 installed. 

If someone can point me in the right direction, it would be much appreciated. 

Hey there, just in case you still have this issue, I had to do this exact same thing to enable coolbits in xorg.conf... Essentially you need to add a fake display in the conf file and add it to the card you're trying to enable coolbits for. The driver refuses to allow coolbits unless it thinks your card is driving a display (for wtfk reasons).

There are 3 ways I know could work of doing this:

- Extract/copy the "edid.bin" data from a monitor you have and create a fake monitor in xorg.conf pointing to this file (google can help. This worked for me on Fedora, I had to go this way 'cause my card doesn't have video outputs).

- There's a package in some distros called "xserver-xorg-video-dummy" or something like that. It might help.

- There are "hdmi dummy" plugs which you can plug into the hdmi port on your card to trick it into thinking there's a display. They're very cheap on amazon/aliexpress (I haven't tested this for this purpose but it saved me when teamviewer just shows a black screen if there's no connected display. It's essentially the same logic so it "should" work).
 

[Gorgon]

3 hours ago, K3Rn3L_P4n1c said:

Hey there, just in case you still have this issue, I had to do this exact same thing to enable coolbits in xorg.conf... Essentially you need to add a fake display in the conf file and add it to the card you're trying to enable coolbits for. The driver refuses to allow coolbits unless it thinks your card is driving a display (for wtfk reasons).

There are 3 ways I know could work of doing this:

- Extract/copy the "edid.bin" data from a monitor you have and create a fake monitor in xorg.conf pointing to this file (google can help. This worked for me on Fedora, I had to go this way 'cause my card doesn't have video outputs).

- There's a package in some distros called "xserver-xorg-video-dummy" or something like that. It might help.

- There are "hdmi dummy" plugs which you can plug into the hdmi port on your card to trick it into thinking there's a display. They're very cheap on amazon/aliexpress (I haven't tested this for this purpose but it saved me when teamviewer just shows a black screen if there's no connected display. It's essentially the same logic so it "should" work).
 

Also:

See the sections on "Configuring X-Windows and GPU" and "Advanced Topics | Headless Operation" These should work under Fedora but you may have to "Season to Taste"

[kotovasia]

Hello. I have a few questions. How to check the system for stability and editing errors. I'm using Poroxmox which doesn't have a GUI.

My videocard 1070ti.

[Gorgon]

On 4/2/2024 at 4:48 PM, kotovasia said:

Hello. I have a few questions. How to check the system for stability and editing errors. I'm using Poroxmox which doesn't have a GUI.

My videocard 1070ti.

I assUme you are referring to Linux?

Running Folding at Home or any other intensive GPU Load will test the stability of the system. For CPU testing I usually use

mprime -t

which is the *nix variant of a Prime 95 Samll FFT Torture Test.

 

Editing Errors???