A Look At The CPU Security Mitigation Costs Three Years After Spectre/Meltdown
Surprisingly it took until recently for kernel developers to discover AES-NI XTS performance taking a huge hit from Retpolines. Patches are pending to reduce the indirect calls in the AES-NI driver to avoid this significant overhead for processors relying on Retpolines as part of their Spectre V2 mitigations.
So at least soon the AES-XTS performance will be in better shape for the older Intel CPUs and all tested AMD CPUs once those kernel driver improvements land.
For code compilation speed, there only ever tends to be a difference on the older CPU models.