I know exactly what your are talking about. Every year I take a trip into the Minnesota wilderness. I only take my glasses and only use them when I have to.
After spending two weeks on a lake fishing without my glasses, I find that my vision is actually getting better from being forced to focus at far away objects.
Course, as soon as I get back to civilization, in go the contacts and I'm blind again.