Interesting, it hadn't occured to me about the latency issue with using a hub. I guess I've been running a hub for so long that I dont notice any difference but I'm willing to accept this could be an issue.
Is there any USB controllers that are better than others? Is it possible to install a controller or make changes that can increase the performance of any of the USB ports?

Wow, what a can of worms that questioned opened for me. I goggled a bit looking for any clues. I eventually came back to Wikipedia since I could mostly understand it
http://en.wikipedia.org/wiki/USBI'll try and hit the high points of the entry, what follows is my edited version:
USB was designed to accommodate hubs, no mention I could find about powered hubs. It is a tiered star topology allowing branching with up to 127 devices per host controller. Communication is based on pipes (logical channels). A pipe is a connection from the host controller to a logical entity, found on a device, and named an endpoint. A pipe correspond 1-to-1 to endpoints.
There are two types of pipes: stream and message pipes depending on the type of data transfer. I assume the pipe we are using with our joysticks is a stream interrupt transfer pipe, these are devices that need guaranteed quick responses. An endpoint of a pipe is addressable with its device_address, endpoint_number.
When a USB device is first connected to a USB host, the USB device enumeration process is started. The enumeration starts by sending a reset signal to the USB device, the devices on the bus are assigned a unique 7-bit address.
In USB 2.0, the host controller polls the bus for traffic, usually in a round-robin fashion. The slowest device connected to a controller sets the bandwidth of the interface.
From my understanding of the above, whether a hub is involved or not really doesn't matter since the device is being addressed directly by the host controller. What matters most in getting the best performance out of USB is the fact that the slowest device on the bus controls the bandwidth of the interface, and the number of devices on the bus since it is polled round-robin.
For best performance only have the joystick connected to the hub.
Hopefully some USB engineer can read this and comment one way or another.