The USB bus is not a subset of the PCI bus. The USB bus is a serial bus, not a parallel bus and has its own controller.
Bus contention occurs when one, or more, cores of a CPU need to access memory. Only one core at a time may access system RAM. When one core has to access system RAM, all the other cores have to wait. The arbitration for the bus between the cores is based on the priority of the process. In the case of USB devices, they get top priority as the bus is too dumb to wait to be serviced or there is risk of losing data.
Bus contention also occurs when a bus master, external of the CPU, accesses system RAM. This also blocks the CPU's from accessing system RAM. Most of the time this is not a bad thing, as bus masters are doing work the CPU would have had to do, but a bus master transfer of data to system RAM will invalidate all data in all the CPU core caches.
You are seeing issues, but they are either occuring fast enough you do not notice it, or you have grown use to the slow down.
The USB bus is best suited for small data transfers, like those of a joystick/HOTAS device. The bus has always been a poor choice for moving a lot of data or a continous stream of data.