On every version of Debian I'm running, it was an optional loadable kernel module. So just:

and it won't be loaded....