PCI Express 1.0 vs. 2.0 vs. 3.0 Performance With NVIDIA/Radeon Graphics On Linux

[MagicMyth]

This aligns with a suspicion I've had for a while ever since running a R7 370 over a external PCI Eepress 2.0 on a Thinkpad X230. The performance under windows seems to be far better than under Linux. I always wondered if the eGPU got starved a little more on Linux. It would be great to see a direct comparison with Windows and might show some interesting results if some older AMD GPU with native Gen 1 interface were tested. I'm not yet a premium subscriber so I'll just say pretty please Michael.

[...]

Edit: It's still interesting to me that 25 FPS still run very smooth on Linux while they are completely stuttering on Windows 10. It's just my own observation but I can surely say that the most FPS are not automatically the best experience. When you have the most consistent frame time with less it's a much better measure. And of course image quality also matters.

I've had a similar feeling/experience. Most noticeably under Shadow of Mordor. Window's definitely gets higher frame rates but is constantly jurky and basically unplayable but under Linux the experience was much smoother and totally playable despite around a 15-20fps average difference.

[MrCooper]

That doesn't sound right -

Yeah, quite some revisionism there.

my recollection was that DRI was developed by the Precision Insight folks (PI -> VA Linux -> Tungsten Graphics -> LunarG

I think I corrected you on the last one before. The Tungsten Graphics (TG) CEO (Jens Owen) left when TG was acquired by VMware and co-founded LunarG, but AFAIK nobody else from TG has anything to do with LunarG; the majority of TG folks continued at VMware, many of them are still there (yours truly being an exception).

[Holograph]

Virtualization users interested in using IOMMU like myself will not be happy to hear about how crappy AMD GPUs run with reduced PCI-E bandwidth.

Yet Nvidia still attempts to block IOMMU usage in their drivers. That can at least be worked around.

But geez, I really don't want to give money to either company right now. Not happy with either scenario.

[schmidtbag]

Virtualization users interested in using IOMMU like myself will not be happy to hear about how crappy AMD GPUs run with reduced PCI-E bandwidth.

Since when does anyone have an IOMMU-compatible system that runs PCIe at gen 1.0 speeds? You're making this situation out to be a lot more disastrous than it really is. And as both I and puleglot have pointed out, this situation seems oddly suspicious, considering the FuryX in Windows does not suffer this problem.

This is a problem, but it doesn't seem to be a problem specific to AMD GPU hardware. It's either a problem with the motherboard or the drivers.

[Holograph]

Since when does anyone have an IOMMU-compatible system that runs PCIe at gen 1.0 speeds? You're making this situation out to be a lot more disastrous than it really is. And as both I and puleglot have pointed out, this situation seems oddly suspicious, considering the FuryX in Windows does not suffer this problem.

This is a problem, but it doesn't seem to be a problem specific to AMD GPU hardware. It's either a problem with the motherboard or the drivers.

It's actually link width, not clock speed, that concerns me.

PCI-E 3.0 x4 should be a reasonable enough link but if link width has the same problem as clockspeed does here then it will suck similarly on AMD GPUs and that is what concerns me.

Maybe it's jumping to conclusions for me to make the assumption that PCI-E 3.0 x4 would work similarly poorly but it seems much more likely true than not.

I give no free passes to any company for any problem that was already solved by a competitor to that company. Regardless of how Windows compares, Nvidia doesn't seem to suffer this problem and I have no problem blaming AMD here. It's not unreasonable to expect a product to work, even in edge cases, even if not everyone has the same expectation.

If you guys can show the problem is elsewhere... feel free. But I'm not suspicious about this at all currently.

[schmidtbag]

It's actually link width, not clock speed, that concerns me.

I wasn't referring to clock speed; I was referring to bandwidth-per-lane. Each generation of PCIe offers roughly double the bandwidth without adding more contact points or changing the bus frequency (which tends to remain around 100MHz).

PCI-E 3.0 x4 should be a reasonable enough link but if link width has the same problem as clockspeed does here then it will suck similarly on AMD GPUs and that is what concerns me.

Even on Windows with an Nvidia GPU, PCIe 3.0 at x4 is shown to be insufficient under certain loads. Expecting it to keep up with x8 or x16 would be unreasonable; why else do you think newer generations of PCIe are released if all we ever needed was the performance offering of the first gen?

I give no free passes to any company for any problem that was already solved by a competitor to that company. Regardless of how Windows compares, Nvidia doesn't seem to suffer this problem and I have no problem blaming AMD here. It's not unreasonable to expect a product to work, even in edge cases, even if not everyone has the same expectation.

That's ridiculous - there is solid evidence that this can be fixed and/or is not a universal problem. Blaming AMD for this is like blaming Lays for stale potato chips because your local grocery store decided to stock expired bags. It's not like AMD regularly tests their GPUs on 10-year-old motherboards...

Assuming fglrx even supports the Fury (not sure if it does) it wouldn't surprise me if even that doesn't suffer this weird performance issue.

[Holograph]

Even on Windows with an Nvidia GPU, PCIe 3.0 at x4 is shown to be insufficient under certain loads. Expecting it to keep up with x8 or x16 would be unreasonable; why else do you think newer generations of PCIe are released if all we ever needed was the performance offering of the first gen?

I don't expect x4 performance to equal x8. But I do expect it to be well over half the performance. 10-20% performance difference would be acceptable.

Similarly, I wouldn't expect PCI-E 1.0 performance to equal 2.0 or 3.0 at the same link width, but it shouldn't be THAT much lower.

I'm willing to look at more evidence on this but I'm not willing to disregard this evidence.

[schmidtbag]

Similarly, I wouldn't expect PCI-E 1.0 performance to equal 2.0 or 3.0 at the same link width, but it shouldn't be THAT much lower.

I'm willing to look at more evidence on this but I'm not willing to disregard this evidence.

As I mentioned before, it's almost exactly half when using gen 1.0. This suggests either the motherboard is doing something sketchy or the open-source drivers weren't developed to compensate for such a huge loss in bandwidth. Unless Michael has a poorly-made card, it is evident that the GPU hardware is not the issue. If the motherboard is the problem then this discovery can mostly be dismissed (that, or you can "give no free pass" to MSI) since other boards don't have this problem. If the drivers are the problem, it is now drawn to the attention of the devs. As long as they take this problem seriously, this is only a temporary issue.

Based on the evidence known, I'm betting the drivers are the problem. But here's what I don't get: Suppose the drivers really are the problem and you stick by your "no free pass" standards. Why is this where you draw the line? There are plenty of other features missing, incomplete, or under-performing on AMD hardware in Linux. Even when looking at strictly Windows, Nvidia hardware has features AMD hardware doesn't, and often outperforms AMD hardware in terms of performance-per-watt. These are things the competition has/does that AMD doesn't. Yet losing performance on a high-end GPU using a decade old specification is what bothers you?

[bridgman]

As I mentioned before, it's almost exactly half when using gen 1.0. This suggests either the motherboard is doing something sketchy or the open-source drivers weren't developed to compensate for such a huge loss in bandwidth.

What is the connection with open source drivers ? AFAIK the tests were run with the AMDGPU-PRO stack...

[schmidtbag]

What is the connection with open source drivers ? AFAIK the tests were run with the AMDGPU-PRO stack...

Correct me if I'm wrong, but isn't AMDGPU-PRO dependent upon AMDGPU (which is open source)?
Pretty much all these tests were telling the same story, and to my understanding, not all of them are utilizing the pro stack, or at least they can be. Regardless, the pro stack is (again, to my understanding) all userland. Wouldn't that be too high-level for that to affected the performance at a hardware-level change? I would think the kernel-level drivers are responsible for communication with the PCIe bus.

Even if I'm wrong about how the drivers work, that ultimately doesn't change my point: this problem is possibly linked to something regarding the Linux AMD drivers. My understanding of what driver specifically is the problem isn't all that relevant.