It sounds to me like your stick is getting tired. If it doesn't return to center properly you'll "be out of trim". When you re trim, do you find its a nose up/down adjustment, or a rolling left/right adjustment? I'd try adding some "deadband" space to both the x and y axises.
With combat trim on, my planes flys pretty much strait and level all the time. The combat trim automatically retrims the plane for what ever speed your running. If you are NOT using combat trim then yes you will be retrimming the plane as you speed changes.
There are two schools about "combat trim". Some believe that triming the plane themselves gives them an edge, while I know a good number of top "killers" in this game use CT. Its just how much "fiddling" you want to do while flying.
Check those two things before you give up on the game.
