Crappy sound card? How are you measuring the CPU usage? Does it take into account IRQ overhead (that is where USB really is piggish)? Decoding music files is very CPU intensive. Nevermind the actual playing of them.
The USB bus is a dumb serial interface. It has absolutely no CPU or hardware assistance at all.