Go into the Map Controllers menu. Sit and watch the numbers, with your hand off the stick, in parentheses next to the axis inputs.
Any one of the axis inputs numbers shifting around would prevent the vehicle from shifting out of gear. Select the axis, check the "Advanced" option. Now add a little deadband. You should be able to see the effect in the graphs of that panel.