The thing that bothers me is, many people claim they do not have this problem. Just today I spoke to somebody on IRC who uses similar software. He's using the 180 nvidia driver with default xorg.conf settings. That is what lead me to believe that it is a problem with nvidia. But on the other hand, my older computer (Celeron 2GHz) has an ATI card in it and the situation is even worse. So it's really confusing and I would love to get to the bottom of this.
It's why people claim Linux is slower than Windows, when in reality it's not - for example Mathematica on Linux calculates twice as fast than on Windows on my computer.