AMD Preparing PCIe TPH Support For Upcoming CPUs

A new patch series sent out today by AMD Linux engineers confirm that PCIe TPH will be supported with "upcoming AMD hardware" as a nice performance optimization feature for PCI Express.

The PCI Express TLP Processing Hints (TPH) are hints that can be injected for improving latency and lowering traffic congestion when there are several possible cache locations in the system. The TLP Processing Hints can note the optimal location of a Transaction Layer Packet (TLP). Sent out today were a set of Linux kernel patches from AMD for implementing PCIe TPH and confirming that this will be supported with upcoming AMD hardware... Given the timing of these patches and AMD tending to not work on Linux enablement multiple generations out, this is presumably for Zen 5 processors. Whether it's for all Zen 5 processors or limited to AMD EPYC Zen 5 processors where it will be most beneficial isn't clear from today's patches.

The AMD patch series cover letter further explains:

"TPH (TLP Processing Hints) is a PCIe feature that allows endpoint devices to provide optimization hints for requests that target memory space. These hints, in a format called steering tag (ST), are provided in the requester's TLP headers and allow the system hardware, including the Root Complex, to optimize the utilization of platform resources for the requests.

Upcoming AMD hardware implement a new Cache Injection feature that leverages TPH. Cache Injection allows PCIe endpoints to inject I/O Coherent DMA writes directly into an L2 within the CCX (core complex) closest to the CPU core that will consume it. The technology is targeted at applications whose performance is sensitive to the latency of inbound writes as seen by a CPU core. The applications include networking and storage applications.

This series implements generic TPH support in Linux. It allows STs to be retrieved from ACPI _DSM (defined by ACPI) and used by PCIe end-point drivers as needed. As a demo, it includes an usage example in Broadcom BNXT driver. When running on Broadcom NICs with proper firmware, Cache Injection shows substantial memory bandwidth saving using real-world benchmarks."

Exciting feature! There are 9 patches out for review implementing the kernel-side functionality. Hopefully it will be upstreamed soon in time for that upcoming AMD hardware with the PCIe TPH support. With this patch series besides the core PCIe TPH enablement in the Linux kernel, only the Broadcom BNXT "bnxt_en" network driver is initially modified for making use of PCIe TPH support but hopefully other driver adaptations will come in time now that hardware support will be rolling out.