Some objects like the flags and and trains don't really fall under the normal down time system.
Flags are destroyed by the system when enough town buildings are destroyed, the dead object then appears white, when the town regens past its capture % the system restores the flag object.