I couldn't duplicate this, on my system the rudder stops when I release the button.
Then again, my keyboard is plugged into the PS2 port, USB may cause different issues especially if there's any chance of power issues I.E. many controllers sharing the same USB host controller.