There may be one negative. (Or one positive to keeping the house)
Our neighbors have a house on their land that is not livable. They are keeping it anyway because of the grandfathering clause.
If they tore it down, and then decided later to build, then the utilities, water line, and septic...