I sort of do it a similar way Anarchy.
I fly with it off, but have a button mapped on my throttle to flip it on/off quickly, just to get a "quick trim" when I slow down or accelerate rapidly, or get trim out of shape from doing anything else as well. I always have it off in a fight once I get slow especially as well, as like you said, when the flaps come out, you end up having to push the nose down often to get lined up for a shot, which is very counter intuitive to always "pulling" for a shot, at least in most cases. Plus when you "pull" for a shot, you can let go a little and the nose tends to stay, where as with CT on and flaps out, when you "push" for that line up, if you let off just a little, the nose takes off going up again in a big way, even with just a slight release it push pressure. That's my experience with it anyhow. With CT on and flaps out, it just feels...screwy.
Once I take off, I usually flip it on for general tooling about, and again, once I'm in a fight and bleeding E in any way, I'll flip it off and use the hat I have mapped for trim on my stick (ail and elevator). Since I don't have rudder trim mapped, and don't find it to really be that much of a big deal for me anyhow, if I need to quickly re-trim while in combat, I'll just hit that mapped throttle trim button once, which does if far faster than holding down the stick trim buttons, which are good for precision IMO, as well as the absolutely required elevator trim for flying the K4 and getting out of control lock up in dives.
I think they way you're doing it (much the same as mine), is the optimal way to have your controls set up. Having CT off in the game settings, and having NO on/off CT or auto-trim button mapped, isn't a great idea IMO, even if you have all the trims mapped to your HOTAS, as having the option of flipping CT on/off is faster in a fight than doing it the manual/fine tune way, just a quick button press and you're good to go, even after a change of a couple hundred MPH in a handful of seconds from a steep dive, climb, turn, or combination of the above. Just my opinion, but it's worked for me for a long time, and some of the "great" K4 and G10 sticks back in the day taught me to do it this way, as there is NO plane where trim is more important - and since you're one of the current K4 pilots like the above - probably best off doing what you're doing.