AMDGPU Linux Driver No Longer Lets You Have Unlimited Control To Lower Your Power Limit

[Anux]

I run my Vega64 with a 45W TDP generally.

Oh shit that's way to low, your luky it didn't explode!!11! But I hope you know that everytime you do this, someone at AMD gets hurt and crys for weeks, how cruel of you.

BTW: does it still work with kernel 6.7?

The way power1_cap_min used to work was to let the gpu do whatever it wanted up to the set wattage, so the user did not have to know the clock, power mode, voltage, etc.

And it still works that way, just not in a useful range (-6%).

People in the bug report mentioned you can set SCLK

That would be a fixed frequency and your power would vary with different workloads.

In the bug reports I also read people using tuxclocker and corectl, for people using these "front ends" would it be possible for the presentation to stay the same, but then the software doing these other methods to set the desired tdp?

No we don't have access to the sensors that the chip/firmware uses to do it's power management. Even if we had access the process would probably be too slow (read sensors from GPU -> calculate power management on CPU -> set FID/VID on GPU). If in that time a part of the silicon overheats we could not react to it fast enough.

[s_j_newbury]

No we don't have access to the sensors that the chip/firmware uses to do it's power management. Even if we had access the process would probably be too slow (read sensors from GPU -> calculate power management on CPU -> set FID/VID on GPU). If in that time a part of the silicon overheats we could not react to it fast enough.

If they are advocating a daemon doing this in userspace (setting manual pstates according to load, and guessing internal state) then I hope they'll offer RMA on melted cards!

[yump]

How so? A continous curve is nothing more than infinite F/V-pairs. If any of those infinite F/V-pairs destroys your card how do they make sure this doesn't happen with some random workload that uses this F/V-pair at standard TDP? I could limit my games FPS from 312 to 1 and achive any possible TDP far below standard TDP.

Suppose the parameters are a high-order polynomial or something. I'm not saying it does work that way, or that this is a legitimate problem -- my only card is Polaris, which has a mere 7 discrete states. What I'm saying is that it is *conceivably* a legitimate problem.

And what about booting with amdgpu.ppfeaturemask=0xffffffff or any other combination? If that is allowed without kernel recompile there is no argument for limiting the min TDP.

No, as long as there is amdgpu.ppfeaturemask, overclocking, fancontrol and under volting it's not. Heck I could wrap my card in a towel and call AMD support to tell them about my over heating issues.

Yes they can do that, if we get our control back it's totally fine by me. Taint the kernel if there is an altered UEFI setting, taint it if dual boot is detected, taint it on full moon, I couldn't care less.

Yeah. IMO, what they should do is:

1. Lower and upper power limits are obeyed and read-only without the overdrive bit set in amdgpu.ppfeaturemask, along with the sysfs files that change voltage/frequency relations.

2. With the overdrive bit set, they are writable.

3. If any of them are written, that tains the kernel.

The reason for tainting only on write is that I have amdgpu.ppfeaturemask in the ansible playbook I run on all family machines, but changing that would only be a small hassle.

IIRC, the status quo is that the overclocking files are writable but do nothing without the overdrive bit, which is... less than optimal I think.

No it doesn't. For fixed frequency under volting decreases current. And if power management selects a higher state, than that's not under volting and it would still only stay at roughly the same current we had before under volting. That is how we get higher performance at same TDP with under volting.

Undervolting, as it is practiced in 2024 (probably since 2016 really), is almost never fixed frequency.

And when power management selects a higher frequency, it will do so constrained by the power limit. P=IV, so current increases. It is not roughly the same current.

Sorry for being a bit too sarcastic. :/ Until recently there was no doubt that my next GPU would be an AMD card, now that I know I couldn't use it in Windows (there's no way I'm running a card above 150 W for extended periods) I have lost faith in them and am seriously considering Intel GPUs.
I think I got their memo, they hate power efficiency and custom builds and want us to buy competitor cards. Intel seems to have no problem going down to 95 W in Win: https://www.kitguru.net/wp-content/u...-oc-scaled.jpg and I noticed they are really working hard to make their drivers good/usable. Maybe 2 more generations and we won't ever have to think about radeon again.

I can only hope this doesn't spread to their CPU lineup.

Agreed. I'm using hybrid graphics with my monitors connected to the motherboard, so I don't care too much about idle power. The GPU can just turn off. So Intel's embarrassments on that front are no trouble to me. And with the rapid progress on NVK, Nvidia is starting to look quite enticing.

It shouldn't be forgotten how much Valve, Collabora, et. al. have contributed to FOSS support for AMD graphics, or that when AMD was working alone what we got was... Catalyst.

Correct. I guess what I meant in the first reply is more in the sense that I think I never in my life heard anyone talk about amperes in this context. Or as I stated somewhat inflammatory, we wouldn't say like he did that current has to increase, we'd say that at iso power and lower voltage f or C is higher, or both (so I guess an example would be that we would see GPU usage being higher at the same tdp and for approx. same amount of work [which could be frames or hashes or w/e]).

Amperes are very important in the context of reliability, and amperes are what cause stress on the VRM.

Almost certainly it's f that varies. The power management controller could theoretically tell the scheduler to idle CUs or parts of CUs, but I'm pretty sure reducing frequency (and voltage, because its a lookup table or parametric curve) saves more power per unit performance lost unless you've set a really low power limit on a large chip.

The question I have now is: is it still possible to limit tdp, but in a less transparent way?

People in the bug report mentioned you can set SCLK, after talking to you guys I looked the documentation and there apparently you can set VDD curve, pp tables and acrivate/deactivate performance levels, and set manual performance levels.

There may be something you can write to some sysfs file or another that would have the effect of restricting GPU frequency. You could have a daemon use feedback control of the average power with that lever to implement a power limit less than the minimum from userspace.

I don't know how fast the PMU firmware's power control loop is, but a userspace software loop would certainly be fast enough for efficiency/noise control (you don't want to go sub-frame anyway) and certainly faster than the thermal inertia of the heatsink. There's no need to worry about overheating the chip because you'll always be backstopped by the regular power and thermal limits.

It would be less reliable than the firmware, however, and I've had a lot of trouble getting my RX 580 to go into really deep power saving states (below D0) if anything is poking around with the sysfs files. I have to disable all of corectrl's monitoring, and my own system fan control daemon goes hands off the GPU if it sees utilization dip below a threshold.

[Anux]

Suppose the parameters are a high-order polynomial or something.

I'm not sure what the exact shape of the curve has to do with anything, my argument doesn't change even if it was a linear function. You could represent any curve with infinite F/V-values.

And when power management selects a higher frequency, it will do so constrained by the power limit. P=IV, so current increases. It is not roughly the same current.

You need to take into account that the frequency changes if you select a higher F/V pair. That changes the resistance (impedance) of the silicon (given a fixed workload) which in turn limits the current at a given voltage. Remember the formula DumbFsck posted above.

[bosslog]

On my RDNA2 card (RX 6650XT) the power limit just makes the driver decrease the frequency/clocks until the power usage is below the limit. The core clock on my card can go from 700 to 2765 MHz by default. The default power limit for my card is 164 W, so when playing a game the driver will increase the frequency as high as it can until the power usage hits 164 W. Heavier games max out the 164 W with a lower frequency (e.g. 2600 MHz) while lighter games will run at the max 2765 MHz while only consuming e.g. 130 W. FurMark can even get to 164 W at just 2400 MHz. If I set the power limit to e.g. 130 W, the maximum frequency will be lower, in the exact same way as if I've set a lower max frequency through OverDrive.

The dynamic adjustment of the frequency to stay below the power cap is completely normal (default!) driver behavior. There are no out-of-spec changes to voltages or currents with decreased power limits; in contrast to under/overvolting, overclocking, and increased power limits! And if you set the power limit at something ridiculous like 0 W, then the driver will just keep the clocks to the minimum allowed without any issues (actual power usage then is e.g. 40 W).

AMD is probably just lying about "potential hardware damage" to protect future GPU sales, as a -20% power cap will significantly extend hardware life in exchange for a minor performance loss.

For those who want a workaround without recompiling the kernel: you can set your maximum shader clock a few hundred MHz lower than default through OverDrive, but you'll lose a bit of performance on light games that would normally max out the clocks while staying at lower power usage.

[yump]

I'm not sure what the exact shape of the curve has to do with anything, my argument doesn't change even if it was a linear function. You could represent any curve with infinite F/V-values.

Polynomial approximations are known for curving toward infinity or negative infinity outside the range they are intended to approximate.

Like, it's conceivable that for frequencies << typical furmark@TDP, applied voltage might actually go up as frequency goes down, because the manufacturer didn't care to tune that part of the curve and it only ever experiences near-zero utilization. Perhaps power-vs-frequency is actually non-monotonic if you go low enough, and perhaps the power limit control loop goes nuts on some cards.

But maybe it's not that kind of curve; I don't know. I don't have anything newer than Polaris, and until/unless this matter is resolved I'm not inclined to chuck any money AMD's way.

You need to take into account that the frequency changes if you select a higher F/V pair. That changes the resistance (impedance) of the silicon (given a fixed workload) which in turn limits the current at a given voltage. Remember the formula DumbFsck posted above.

Yes, that is exactly the reason undervolting increases current if the PMU is allowed to do as it will. I = fCV is an impedance-shaped relation. Rearrange the formula and you get V/I = 1/(fC). Higher frequency -> lower impedance.

[Phoronos]

AMD did right here : you shoud NOT be able to play with the voltage or the amperage of the hardware, it could destroy it very fast.

[AmericanLocomotive]

I'm not sure what the exact shape of the curve has to do with anything, my argument doesn't change even if it was a linear function. You could represent any curve with infinite F/V-values.

You need to take into account that the frequency changes if you select a higher F/V pair. That changes the resistance (impedance) of the silicon (given a fixed workload) which in turn limits the current at a given voltage. Remember the formula DumbFsck posted above.

Modern GPUs are very complex devices, with many different clock and power domains. You're approaching this as if the GPU is a single monolithic device, and all of the components run in lockstep, and can run at any possible frequency.

That simply isn't true.

There are components inside the GPU beyond just the shader cores that need to run at specific speeds at all times to do their jobs. The display engines have to run at a certain speed in order to push pixels out through the display ports fast enough. The memory controller needs to run at a high enough speed to adequately feed framebuffer data to display engine and keep the graphics memory alive. The video decoder/encoder block has its own specific clock domain and speed it runs at separate from everything else. The PCI-E PHY has a specific speed it needs to run on to stay synchronized and active on the PCI-E bus.

Plus, most (if not all) modern GPUs use clock & power gating to achieve their idle clocks. So when you check your GPU stats at idle, and you see "75 MHz, 9 watts" - that's not actually true. The shader cores are waking up, doing some work at a much higher speed (say 400 MHz) at a high power (say 25w), and then going back to sleep, and then power-gated off. So over a short time interval (say 1-2 seconds), the average speed and power will be 75Mhz and 9w - but the instantaneous will be far higher.

Since those components all have fixed speed (or strict minimum speeds) they need to run at it to ensure stability and basic functionality of the card - that means they have minimum voltages that HAVE to be met. That by definition means the card has a bare minimum lower power limit. This power limit can very based on specific card configuration, silicon bin, and perhaps even accessories. I know on nVidia cards, things like fans and RGB lights can count towards card power limits.

So if the card is programmed with a parametric/equation-based F/V curve, this can lead to severe issues if you allow users to set arbitrary power limits. Let's say your card idles at "9w" reported. You want it lower, so you set the power limit to 7w. System immediately crashes, as the display engine and memory controller have hardware minimum clock limits. The driver is forcing the 7w power limit, so the VRM has no choice but to force the core voltages below the F/V curve, and the display engine crashes.

Another more realistic example: The card idles at "9w" according to the software telemetry. In basic desktop use, you never see the power consumption go above 15w. You decide you want to cap the card to "15w". You set the limit, everything seems fine. The GPU core has gone to sleep since nothing is changing on the screen. The display engines are just running in a repeat/auto-refresh mode since nothing new needs to be rendered. You wiggle the mouse, and suddenly the GPU crashes. The GPU core woke up to render your mouse movement. The shaders have a minimum clock speed they have to operate at. The GPU immediately plowed into the 15w TDP limit in a manner of microseconds, the VRM immediately cratered the vCore trying to meet the 15w limit imposed by the driver, and then the GPU core crashed due to low voltage.

Remember, the power readings, temperatures, core speeds, etc... you are reading through the OS are just averaged and filtered values reported by the driver. At the actual hardware level, the power, temperatures, clock speeds, etc... are being sampled hundreds, if not thousands of times per second.

The main impetus for this, is that people will set power limits too low, and will crash GPUs. While it seems unlikely, I believe that a driver-power-limit-imposed-momentarily-voltage-hiccup could cause "SCR latchup" inside the core. That's a situation where transistors that would never normally be on at the same time, could both be triggered, causing a short circuit.

Yeah losing some control over the hardware stinks, but the reality is that being able to adjust GPU power limits at all through software is still a "relatively" new thing. It wasn't that long ago the only thing you could adjust was clocks and voltage - and even that wasn't always a given.

[Elipsus]

[QUOTE=bosslog;n1448693

AMD is probably just lying about "potential hardware damage" to protect future GPU sales, as a -20% power cap will significantly extend hardware life in exchange for a minor performance loss.

That is a very far fetched statement, having a card die of old age isn't common....at all, even on normal power limits, and even on overclocked cards.....

And even if that was true, lets do some napkin math :[LIST][*]linux is 4% of the OS market share[*]AMD is less than 25% of the GPU market (even less if we put the people only having iGPU in the equation)[*]probably less than 5% of the users will ever reduce the power limit (I'm being extremely generous here)[/LIST]so we're very optimistically talking about 0.05% of GPU market potentially having a longer lasting hardware, and I won't even try to estimate how many of these will actually wait until their GPU dies out to replace it, because that number is probably way too close to zero to make sens
So, AMD made this choice to have a, in absolute best case scenario where everyone wait until their card dies and buy again an AMD card increase in sales of a whooping 6350 Cards per year (out of the 12 700 000 dGPU sold per year total).

So, no, they didn't do this for the increase in sale.

[yump]

AmericanLocomotive I'm pretty sure the power limit acts over much longer timescales than that, based Igor's power spike data. https://www.igorslab.de/en/radeon-rx...0xt-review/11/

There are physical limits to how quickly VRM output voltage can be adjusted, and it would be extremely weird for a power limit to be used as anything other than an outer control loop shrinking the allowed range of frequency selection. If it could extend the range below Vmin, or force unsafe combinations of frequencies from different domains, then you'd have to validate the power limit against all possible workloads. Better to write the firmware so that it does not do that, and you can list the constraints to be obeyed explicitly.

Intel CPUs, for example, will violate the power limit to maintain base clock, but they won't violate the thermal limit to do that. And if they get all the way down to the minimum frequency (800 MHz) and are *still* overheating, then they switch to clock modulation instead (forced idle-injection on a duty cycle). But at no point will they attempt to run below fused minimum frequencies and voltages. They'll emergency shutdown first.

Yeah losing some control over the hardware stinks, but the reality is that being able to adjust GPU power limits at all through software is still a "relatively" new thing. It wasn't that long ago the only thing you could adjust was clocks and voltage - and even that wasn't always a given.

I don't know about the Nvidia side, but power limits have be software adjustable on AMD GPUs since at least polaris in 2016, maybe earlier.