The rudder's trim rarely hampers flying as much as the elevator's, in my experience. Only in planes with lots of torque or that get very sideways (p-factor I think) like the Il2 and F4U and 152 does it really pay off to spend the time trimming the rudders. Most of the time you're just as well (if not better, since it's faster) simply pushing your pedals or twisting your stick to get the nose where you want it. Trim is a slow thing, so it's only really attractive for comparably slow situations.. Like if you plan ahead of time to make a move where the bit of extra control authority will make a difference (and again this small a difference can be found in e.g. merely reducing your action/reaction time by a few fractions of a second), or if you've got a twisty stick that's quite a bit more bothersome to twist one way or the other.
It's really up to you.