1) Reset all the advanced parameters back to defaults.
2) Calibrate the stick in the game, making sure to get each axis.
3) Once that is done, go into "Options->Controls->Map Controllers". Check the numbers in parantheses, next to each axis. With your hand off the stick, those numbers should not be changing. If they are, make note of the axis. Now, move each axis to its fullest deflection. The numbers for each axis should go from 0 to 65535. If they do not, then calibrate that particular axis. If they still do not, then calibrate the stick in Windows.
By the way, the numbers in parantheses are the numbers being sent to the game from the stick.
Now, assuming all is ok up to now. For each axis that showed numbers changing when the stick was not being moved, select that axis, then check the "Advanced" option. In the "Advanced" panel, add a little bit of deadband (a little goes a long way) and check the results in the graphs in the "Advanced" panel. You should see them smooth out.
Repeat as neccessary for each axis.