You are getting too involved with them and possibly making them feel like they have a say on what you do with YOUR land.
Follow the land restrictions, the local laws, etc and then do what you want. Keep your place looking nice and tidy. Be respectful of them and nice but at the same time do what you want with your land. DO NOT use their land to access your land in anyway.
Even if they get mouthy or mean keep your cool and be nice. They will eventually settle down when the changes become permanent. People don't like change.
The key is... If they didn't want the land to change they should have purchased it!