That's what my dad thought when they bought the house when he retired. Now Dad's 81 and Mom 89 and they just moved before Christmas. Instead of buying they rent an apartment to avoid hassle when either of them dies. A wise move considering there'll be no more lawn mowing, snow shoveling, maintaining the house in proper condition...
That is why we built a two story duplex. We rent the downstairs (small 1 bedroom/1 bath) out so it pays for taxes and maintenance. Then when one of us croaks, the remaining one will move downstairs and rent the upstairs out.
Speaking of maintenance. There is no wood on the outside of the house for termites or rot (HardiePlank). I had the house painted in a UV stabilized urethane. Should be good long past our deaths. We don't have snow to worry about. The rental pays for lawn maintenance (tax write-off as well).
We hired a property management company to handle the rental so we just collect the checks. They take care of making sure it is rented and maintained.
Barring the city invoking eminent domain (not likely as the house is in a national historic district) we should not have any issues.
We are looking at picking up a couple more duplexes in the area.