dcraw slowed down when it came to using kernel-debug.
It is the same story with SQLite.
Ending off our testing, the kernel-debug package led to a massive performance drop when testing the PostgreSQL performance using pgbench. Disabling SELinux / Audit led to about a 5% gain.
Well, the area where SELinux / Audit and the debugging-enabled kernel really impacted the performance was with the disk and database tests (along with Apache). In the other Linux desktop benchmarks, there was a smaller margin, with some being more noticeable than the others. Disabling SELinux and Audit will certainly improve the performance of Fedora, while running a kernel with all of the debugging code enabled will cost you quite a bit in the way of performance. For developers, having this kernel debugging support is important, while for security-oriented users, having Security Enhanced Linux and system-call auditing support is important and worth the low cost, even with Intel Atom hardware.