AMD Smart Access Memory / Resizable BAR On Linux Still Ripe For Improvement

[Kano]

@bridgman

There is often a marketing and a technical side of the same feature. The product presentation was 100 % marketing for AMD. As the feature is technical possible with much more systems then an override to try it should be there - no matter which OS is currently used.

It is nice to have see that there are simple ways to enable it on "unsupported" hardware - I really like this.

But there are much more examples where marketing affects the features which are available, basically everyone got already used to. OC only possible for some CPUs and motherboard combinations are pure marketing, the chips itself are identical. Limiting some chipsets to lower PCI-E speed is the same. Then all the artificial differences between the FireGL and Radeon or Quadro and GeForce series for major OpenGL apps - there have been patches/hacks out there to flip just a small flag and performance was increased. This could be extended to many more examples like ECC support. Marketing is certainly needed and a "professional" or "gaming" line of products increases the revenues because you have to pay more to flip on things which are technically always possible. I can understand it but I basically dislike this behaviour.

[artivision]

Yes, my post was inappropriate. I already gave my apology.



Oh yes they most likely will. If you look at the patches added to the Linux drivers, this allows removing unnecessary copies and buffering in system RAM. The driver can now directly output command buffers, etc., into GPU memory. This is something every GPU driver would benefit from.

Nope, i test it with Navi1, Navi2 and Polaris. The first two benefit but Polaris 1% low is half the normal. This future may be not optimal for all hardware. Some hardware may work better with its own path. Nvidia is probably that hardware, next two months we will see Nvidia to take the down road.

[duby229]

New results (switch from 3800xt to 5600x)

b550 aorus pro, 5600x 6c12t (oc at 4.6ghz), vega 56 stock, linux 5.10.1 mesagit 13 dec 2020.

With my new 5600x the bios showup the menu: "re-size bar support" just after menu "above 4G decoding", with the ryzen 3800xt it was not listed.

bios csm: off, above 4G decoding: off, re-size bar support disabled:
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=256M

$ vblank_mode=0 glxgears
156142 frames in 5.0 seconds = 31228.395 FPS
155482 frames in 5.0 seconds = 31096.229 FPS
158799 frames in 5.0 seconds = 31759.801 FPS

Unigine Heaven 4.0:
1920x1080 ultra-extreme-8x
fps: 72.5
score: 1826
min fps: 12.8
max fps: 144.9

Counter-strike global offensive:
2560x1440 quality lowest: 514 fps


bios csm: off, above 4G decoding: on, re-size bar support auto (enable are not listed, gigabyte use auto instead enable):
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=8192M

$ vblank_mode=0 glxgears
116654 frames in 5.0 seconds = 23330.732 FPS
116385 frames in 5.0 seconds = 23276.818 FPS
112109 frames in 5.0 seconds = 22421.645 FPS

Unigine Heaven 4.0:
1920x1080 ultra-extreme-8x
fps: 72.5
score: 1826
min fps: 12.6
max fps: 145.4

Counter-strike global offensive:
2560x1440 quality lowest: 516 fps


Not benefice for my use for now.
glxgears show 1/3 less fps, maybe have also negative impact elsewhere i dont know.



Before with the ryzen 3800xt i was having this:
# dmesg | grep BAR
[ 0.280330] pci 0000:0c:00.0: BAR 0: assigned to efifb
[ 4.155501] amdgpu 0000:0c:00.0: BAR 2: releasing [mem 0x7ff0000000-0x7ff01fffff 64bit pref]
[ 4.155504] amdgpu 0000:0c:00.0: BAR 0: releasing [mem 0x7fe0000000-0x7fefffffff 64bit pref]
[ 4.155523] pcieport 0000:0b:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155526] pcieport 0000:0a:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155528] pcieport 0000:00:03.1: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155537] pcieport 0000:00:03.1: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155539] pcieport 0000:0a:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155542] pcieport 0000:0b:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155545] amdgpu 0000:0c:00.0: BAR 0: assigned [mem 0xa00000000-0xbffffffff 64bit pref]
[ 4.155553] amdgpu 0000:0c:00.0: BAR 2: assigned [mem 0x900000000-0x9001fffff 64bit pref]
[ 4.155637] [drm] Detected VRAM RAM=8176M, BAR=8192M

Now with the ryzen 5600x i have only this:
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=8192M

Same bios version, just going from linux 5.10 to 5.10.1 (and debian bullseye latest updates)

I would just like to point out that glxgears is not a benchmark, just having it placed in a different spot on screen will result in differing values. Glxgears purpose is only as an example to show that direct rendering is working.

Also, I'm at least fairly certain this resizable bar support will show the most gain on workloads that have to move data between system memory and graphics memory, perhaps things like compute workloads. I don't think many games will move data between CPU and GPU too much, so its doubtful whether games will ever see much improvement from this.

[kripteks]

New results (switch from 3800xt to 5600x)

b550 aorus pro, 5600x 6c12t (oc at 4.6ghz), vega 56 stock, linux 5.10.1 mesagit 13 dec 2020.

With my new 5600x the bios showup the menu: "re-size bar support" just after menu "above 4G decoding", with the ryzen 3800xt it was not listed.

bios csm: off, above 4G decoding: off, re-size bar support disabled:
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=256M

$ vblank_mode=0 glxgears
156142 frames in 5.0 seconds = 31228.395 FPS
155482 frames in 5.0 seconds = 31096.229 FPS
158799 frames in 5.0 seconds = 31759.801 FPS

Unigine Heaven 4.0:
1920x1080 ultra-extreme-8x
fps: 72.5
score: 1826
min fps: 12.8
max fps: 144.9

Counter-strike global offensive:
2560x1440 quality lowest: 514 fps


bios csm: off, above 4G decoding: on, re-size bar support auto (enable are not listed, gigabyte use auto instead enable):
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=8192M

$ vblank_mode=0 glxgears
116654 frames in 5.0 seconds = 23330.732 FPS
116385 frames in 5.0 seconds = 23276.818 FPS
112109 frames in 5.0 seconds = 22421.645 FPS

Unigine Heaven 4.0:
1920x1080 ultra-extreme-8x
fps: 72.5
score: 1826
min fps: 12.6
max fps: 145.4

Counter-strike global offensive:
2560x1440 quality lowest: 516 fps


Not benefice for my use for now.
glxgears show 1/3 less fps, maybe have also negative impact elsewhere i dont know.



Before with the ryzen 3800xt i was having this:
# dmesg | grep BAR
[ 0.280330] pci 0000:0c:00.0: BAR 0: assigned to efifb
[ 4.155501] amdgpu 0000:0c:00.0: BAR 2: releasing [mem 0x7ff0000000-0x7ff01fffff 64bit pref]
[ 4.155504] amdgpu 0000:0c:00.0: BAR 0: releasing [mem 0x7fe0000000-0x7fefffffff 64bit pref]
[ 4.155523] pcieport 0000:0b:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155526] pcieport 0000:0a:00.0: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155528] pcieport 0000:00:03.1: BAR 15: releasing [mem 0x7fe0000000-0x7ff01fffff 64bit pref]
[ 4.155537] pcieport 0000:00:03.1: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155539] pcieport 0000:0a:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155542] pcieport 0000:0b:00.0: BAR 15: assigned [mem 0x900000000-0xbffffffff 64bit pref]
[ 4.155545] amdgpu 0000:0c:00.0: BAR 0: assigned [mem 0xa00000000-0xbffffffff 64bit pref]
[ 4.155553] amdgpu 0000:0c:00.0: BAR 2: assigned [mem 0x900000000-0x9001fffff 64bit pref]
[ 4.155637] [drm] Detected VRAM RAM=8176M, BAR=8192M

Now with the ryzen 5600x i have only this:
# dmesg | grep BAR
pci 0000:0c:00.0: BAR 0: assigned to efifb
[drm] Detected VRAM RAM=8176M, BAR=8192M

Same bios version, just going from linux 5.10 to 5.10.1 (and debian bullseye latest updates)

[schmidtbag]

It could only affect the "MTD Kernel Subsystem" method, and that one will be too slow to be usable even with PCIe (direct CPU reads from VRAM are on the order of tens of MB/s, no typo!).

Oh yeah, no doubt it would be too slow over PCIe, but I imagine it would still yield a substantial performance increase, and I think this would be a decent synthetic benchmark of how much potential gain you could get.

[MrCooper]

Makes me wonder how much of a difference SAM would make for this:
Swap on video RAM - ArchWiki (archlinux.org)

It could only affect the "MTD Kernel Subsystem" method, and that one will be too slow to be usable even with PCIe (direct CPU reads from VRAM are on the order of tens of MB/s, no typo!).

[pal666]

If csm is enabled, then above 4g decoding will be grayed out / not available so you can't have both things enabled.

it's your motherboard bios implementation detail, others are different

[user1]

no, you need both "above 4g decoding=enabled" and "csm=disabled"

If csm is enabled, then above 4g decoding will be grayed out / not available so you can't have both things enabled.

[pal666]

Apparently, if you have "above 4g decoding" enabled in bios, that means you have resizable BAR support on Linux/AMDGPU.

no, you need both "above 4g decoding=enabled" and "csm=disabled"

[pal666]

Some clarification

some rationalization from nvidiot