To clarify there are a lot of variants of this:
1. Older platforms that support >4G decoding. This provides an additional MMIO region above 4G that can be used for MMIO. The GPU kernel driver can resize the BARs at driver load time if there is enough MMIO space available. The BIOS does not resize the GPU BAR in this case, the GPU kernel driver does. Some of these platforms seem to have bugs where they set incorrect CPU cache mappings on parts the extended MMIO space. This feature was mainly implemented for mining so there would enough MMIO space on the platform for lots of devices with large BARs like GPUs.
2. Newer platforms that support ReBAR or SAM. This provides an additional MMIO region above 4G that can be used for MMIO and the BIOS resizes the GPU BAR when it enumerates the devices on the platform. In this case, the GPU kernel driver doesn't resize the BAR because the BIOS has already done it. There seem to be fewer BIOS bugs in this case because this feature was specifically targetting GPUs so it's better verified by platform vendors.
Regardless of whether you are using 1 or 2 above, the GPU kernel driver has direct access to the entire amount of VRAM via the PCI BAR. As such, if a UMD requests CPU access to a GPU VRAM buffer, we don't have to worry about migrating to system memory if there is contention for due to having a smaller VRAM BAR. This is always taken advantage of regardless of what the UMD decides to do.
Finally, in addition to all of this, the UMDs also made decisions about certain behavior. These particular behaviors are what the article is about.
1. Older platforms that support >4G decoding. This provides an additional MMIO region above 4G that can be used for MMIO. The GPU kernel driver can resize the BARs at driver load time if there is enough MMIO space available. The BIOS does not resize the GPU BAR in this case, the GPU kernel driver does. Some of these platforms seem to have bugs where they set incorrect CPU cache mappings on parts the extended MMIO space. This feature was mainly implemented for mining so there would enough MMIO space on the platform for lots of devices with large BARs like GPUs.
2. Newer platforms that support ReBAR or SAM. This provides an additional MMIO region above 4G that can be used for MMIO and the BIOS resizes the GPU BAR when it enumerates the devices on the platform. In this case, the GPU kernel driver doesn't resize the BAR because the BIOS has already done it. There seem to be fewer BIOS bugs in this case because this feature was specifically targetting GPUs so it's better verified by platform vendors.
Regardless of whether you are using 1 or 2 above, the GPU kernel driver has direct access to the entire amount of VRAM via the PCI BAR. As such, if a UMD requests CPU access to a GPU VRAM buffer, we don't have to worry about migrating to system memory if there is contention for due to having a smaller VRAM BAR. This is always taken advantage of regardless of what the UMD decides to do.
Finally, in addition to all of this, the UMDs also made decisions about certain behavior. These particular behaviors are what the article is about.
Comment