Setting ulimit (-m) low, the first thing I tried, seems to not solve the problem, on some machines (my old nForce-based motherboard is one of them) it even increases the problem due to forcing swapping much earlier.
I have not been able to fully eliminate these hard drops in repsonsivenes by any means, short of running 32-bit PAE on a machine with much ram, in which case at least single offending processes will get OOM before they trigger swapping, or of course disable the swap which also has it's downsides.
In a multi-user-system, for intentional DoS, personally I do not know a reasonable way to combat it. (Short of disabling swap) For me, ulimit -m doesn't help much, -v hurts even non-problematic cases like mmap. Perhaps there are ways to limit per-user use via cgroups or other quota mechanisms though?
That is why I was interested to see if the behavior is better with these new patches. When investigating, I've got a shoddy feeling that the process scheduler blocks on I/O instead of deferring execution of the swapped out-process, in favor of processes that could actually run now.