Their actions have forced your hand. The surveyor doesn't have to do a full stake survey of all your land, just establish where the property line is with the troublesome neighbor. Better a surveyor mark off that section of land than you. If they have a dispute about the property line, it's not with you at that point, it's with their own title company or whatever.
Also, having a campsite, tree-fort, hunting blind or the like doesn't establish adverse possession. Now, if their driveway has been on a portion of your property for x # of years, maybe you'll have to provide an easement or another driveway for them. But, their recreation typically is not going to persuade a state to take property from one person and deed it to another.