AMD Smart Access Memory / Resizable BAR On Linux Still Ripe For Improvement

[kripteks]

I was disappointed by this at first as well, but turns out "above 4G decoding" is enough for BAR resizing to work in Linux (I had to disable CSM as well though). The SAM specific option is only needed for Windows.

You can check for it working like this:

> sudo dmesg|grep BAR=
[drm] Detected VRAM RAM=8176M, BAR=8192M

If it says "BAR=256M", it's not working.

You right.
csm on: [drm] Detected VRAM RAM=8176M, BAR=256M
csm off: [drm] Detected VRAM RAM=8176M, BAR=8192M

Here results:
b550 aorus pro, 3800xt, vega 56, all on stock, linux 5.10, mesa git 13 dec 2020.

bios csm: off, above 4G decoding: on
# dmesg | grep BAR
[ 0.280330] pci 0000:0c:00.0: BAR 0: assigned to efifb
[ 4.155501] amdgpu 0000:0c:00.0: BAR 2: releasing [mem 0x7ff0000000-0x7ff01fffff 64bit pref]
[ 4.155504] amdgpu 0000:0c:00.0: BAR 0: releasing [mem 0x7fe0000000-0x7fefffffff 64bit pref]
[ 4.155523] pcieport 0000:0b:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155526] pcieport 0000:0a:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155528] pcieport 0000:00:03.1: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155537] pcieport 0000:00:03.1: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155539] pcieport 0000:0a:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155542] pcieport 0000:0b:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155545] amdgpu 0000:0c:00.0: BAR 0: assigned [mem 0xa00000000-0xbffffffff 64bit pref]
[ 4.155553] amdgpu 0000:0c:00.0: BAR 2: assigned [mem 0x900000000-0x9001fffff 64bit pref]
[ 4.155637] [drm] Detected VRAM RAM=8176M, BAR=8192M

$ vblank_mode=0 glxgears
ATTENTION: default value of option vblank_mode overridden by environment.
71851 frames in 5.0 seconds = 14370.048 FPS
76439 frames in 5.0 seconds = 15287.775 FPS
76745 frames in 5.0 seconds = 15348.868 FPS

Unigine Superposition v1.1:
preset high
result: 7915
fps min: 50.64, avg 59.20, max 71.16
GPU °C: Min 43.0 Max 66.0

Unigine Heaven 4.0:
1080p, ultra, extreme, 8x
fps: 73.1
Score: 1843
Min FPS: 11.3
Max FPS: 147.6



bios csm: off, above 4G decoding: off
# dmesg | grep BAR
[ 0.282549] pci 0000:0c:00.0: BAR 0: assigned to efifb
[ 4.249088] [drm] Detected VRAM RAM=8176M, BAR=256M

$ vblank_mode=0 glxgears
ATTENTION: default value of option vblank_mode overridden by environment.
98150 frames in 5.0 seconds = 19629.842 FPS
93457 frames in 5.0 seconds = 18691.219 FPS
94206 frames in 5.0 seconds = 18841.086 FPS

Unigine Superposition v1.1:
preset high
result: 7915
fps min: 50.53, avg 59.20, max 71.68
GPU °C: Min 46.0 Max 65.0

Unigine Heaven 4.0:
1080p, ultra, extreme, 8x
fps: 73.0
Score: 1838
Min FPS: 11.3
Max FPS: 146.5




b550 aorus pro, 3800xt 6 core 4775mhz, memory 3600mhz fclk 1800mhz, vega 56 1590sclk-900-mclk, linux 5.10, mesa git 13 dec 2020
counter-strike global offensive:
2560x1440, low quality
above 4G decoding off: 392 fps, 387 fps
above 4G decoding on: 385 fps, 399 fps

[duby229]

Makes me wonder how much of a difference SAM would make for this:
Swap on video RAM - ArchWiki (archlinux.org)

I forget exactly what chipset it was, but ATi used to have an IGP with dedicated RAM and I did exactly this with it...

[schmidtbag]

Makes me wonder how much of a difference SAM would make for this:
Swap on video RAM - ArchWiki (archlinux.org)

[MrCooper]

I have gigabyte b550 aorus pro with the lastest beta bios ("Add Re-size bar option for AMD Smart Access Memory support").
I dont see amd sam option in bios, i have ryzen 3800xt , the "above 4g decoding" is here.

I was disappointed by this at first as well, but turns out "above 4G decoding" is enough for BAR resizing to work in Linux (I had to disable CSM as well though). The SAM specific option is only needed for Windows.

You can check for it working like this:

> sudo dmesg|grep BAR=
[drm] Detected VRAM RAM=8176M, BAR=8192M

If it says "BAR=256M", it's not working.

[user1]

Apparently, if you have "above 4g decoding" enabled in bios, that means you have resizable BAR support on Linux/AMDGPU. At least with ASUS, not all boards have this BIOS setting. I know that all or most ROG boards have this setting and some Prime boards have it as well, but my board (Prime Z390-P) doesn't have it.

Update: I actually found it in System Agent (SA) Configuration under "advanced" section. Because in other motherboards it is under "boot" section.

[bridgman]

What does BAR stand for?

Base Address Register - it's a register in PCIE config space that sets up an aperture into a PCIE device's on-board resources. On a GPU you typically have one for ROM, one for VRAM, one for GPU registers, and a couple of others I forget...

[shmerl]

What does BAR stand for?

[kiffmet]

Some clarification. On CPU side you require some instructions regarding rBAR to use it effectivly, and those AMD had only implemented by microcode (mostly PDEP/PEXT)...

This has already been debunked by Dr. Ian Cutress

[tajjada]

Could anyone explain it to me as if I was 5 what's the resizable bar etc? Why is it important?

Traditionally, PCIe allowed only a small window of the GPU's video memory to be accessed from the CPU. This limited how the driver can load data into the GPU. It had to prepare things in system memory first and then copy them in pieces.

Now, that window can be resized. This means the driver can just make it as big as the whole video memory, so it can directly access all of it. This simplifies the driver and allows for performance improvements like writing data and commands directly to the GPU.

[piotrj3]

Here's the thing, SAM is more than just resizable BAR [rBAR] support, it's also the OS and driver support on both the CPU and GPU side to take advantage of the fast paths that rBAR support allows for. Although resizable BAR support itself doesn't have any real "special sauce", there is legitimate special sauce here: because AMD controls the CPU and GPU side of things, they can add the optimizations that rBAR support allows for to their driver and ensure/validate that it will work on ALL Ryzen 5000/RX 6000 setups. As seen here, any other setup is kinda a pig in a poke whether the mobo will support turning rBAR on, and even if you do, your OS and drivers may or may not support enabling the applicable fast paths allowed by rBAR support. So yes, SAM theoretically can be supported on many more systems than just Ryzen 5000/RX 6000 combos, but if you want SAM right now on Windows (majority of gamers), you do actually in all probability need the Ryzen 5000/RX 6000 combo. Us Linux nerds can take advantage of SAM on more setups, but we're the exception, not the norm.

Some clarification. On CPU side you require some instructions regarding rBAR to use it effectivly, and those AMD had only implemented by microcode (mostly PDEP/PEXT). So it is not even about particulary validation of some sort, it is simply that Zen 2 and older cannot take actual performance advantage of that. Intel had that since Haswell. In terms of GPUs there is no special hardware treatment, only that driver of GPU allows to take advantage of it. So some angry noises for AMD could come that they didn't apply it to their older GPUs as well. I wouldn't be suprised tho if Nvidia released driver update that supports SAM for all their GPUs still supported.

honestly for Intel stuff is extremly easy, all they have to do is flip the switch on motherboards to say that those stuff are working and existing and if AMD doesn't block them, Intel should be seen as CPUs supporting exactly same things as AMD in terms of SAM. For Nvidia they do simply need to write on driver side stuff that takes advantage of those instructions when they are availiable.