I've just had a very frustrating few days with similar symptoms
on my Mac. I my case this finally turned out to be the sound
card spontaneously setting itself to 41000 samples/sec  despite
me setting to 48000. You can do this with Audio/Midi setup which
is on every Mac.
However, this persisted until I rebooted my machine, when it finally
disappeared.
41000 sample rate definitely will cause a horrible rate of decoding,
although it should be sometimes visible as an evenly spaced set
of spike across the waterfall. At any rate, check you sample rates.

Mike