Sounds like my battle 20 years ago.... My neighbor posted NO TRESPASSING signs on my property and proceeded to claim the property. They also claimed ownership of the road to my property and said I was land locked, but that is another part to the story that I won't go into now. It took many thousands of dollars to defend my claim and I was not reimbursed for any of it through the title company. Now, 17 years later, even after the battle was settled, and the court approved the boundary line agreement, the survey recorded, and quit claim deeds signed by both parties, they continue to violate my land and trespass.
First, I would suggest that you have your survey registered with whatever governmental agency records deeds and survey maps. Then have the surveyor come back and walk the boundaries with you and if possible have a third neutral party with you when this happens, so if you wind up in a court battle, you can bring in the neutral party to testify what the surveyor has said. Then, and only then, remove his signs in front of the surveyor and dispose of them. Also put your signs on all the trees that are closest to the property line and space them approximately 50 feet apart. Many states have a standard for this spacing, but more are better. Then photograph the surveyor and the neutral party at each pin and make certain that the pins are driven into the ground to a point that they can't be removed easily. Many people will either remove a pin or relocate it. Both actions are illegal, but you have to prove who did it to prosecute and that is a difficult task unless you actually see them doing it. Every one of the pins that the surveyor placed on my common boundary with my neighbor has been removed. I know that I didn't do it, but I can't prove who did do it, so I am left with the cost of having them re established. Document everything and don't take anything for granted. People will fight tooth and nail for inches when it comes to land. Hopefully you don't have a deed that is vague such as from the snow pile to the oak tree along the fence line and all these things were there in 1900 and not in 2003.
One thing to consider is to remove every tree that is on your property that is close to the property line so you will have a clear line of sight of the pins. It would be best to have the surveyor there telling you what trees to cut. I have learned a lot in the past 20 years and the one thing that I know is that when someone claims your property as theirs, the only winners in the end are the lawyers.
If you are so inclined (translates to brave), ask the neighbor if he has had his land surveyed and if he has, ask to compare maps. Then if there is a discrepancy, ask the neighbor to have his surveyor contact your surveyor and try to resolve the differences. If the neighbor never has had his land surveyed, and is relying on hearsay, then you are in for a battle unless he can be convinced otherwise.
If you have the time, you can do a lot of the research yourself by looking up the land records and getting a copy of his deed and comparing it to yours to see how the boundary line language compares. Also ask if there is a map of his land on file. Try to go back in the records as far as you can to see if anything has changed on the deeds from transfer to transfer. The more homework that you do the less it will cost you in legal fees. The deeds and maps are public record and aren't very expensive to get copies of them. Start building your file of pictures, and documents now. Keep a note book of all conversations and what was said. If you have a video camera, even better.... make a movie of everything. I am certain that others that have been through this will also have other suggestions too....