Well I am no lawer(but I did sleep in a holiday in last night

). When he says he is going to sue, say "For 3 inches?, Go right ahead" What is the worst you have to loose? Even using his survey, 3" is not much. If you go to court, I would take in any old pics you have of the place showing the history of the fence.
You have 30 years of history showing that fence as being the delinieation of your property and the adjoining lot. And for that length of time, you have honestly believed that to be the accurate line based on the old surveys. 3" surely does not show any intent to defraud, by whoever built the fence. 30' might, but not 3". Heck 3" is probably less than the thickness of the fence.
Oh, and when it is your turn to speak to the judge, I woud start, just how you started here. "My new neighbor approached me to ask me to pay half HIS new fence cost, to which I refused. Then he had his survey, and here we are arguing over 3"..."
It was mentioned before, that existing structures are rarely torn down, that is why it is important to catch transgressors before they get something built and can establish history. Worst case, that is all the judge will order, as there are surely no damages associated with 3" of property use. That is what you would be out if you cave to his threats/demands, so what is the risk? Politely call his bluff, then go have your day in court, if your new "Neighbor" is willing to risk the lawer fees on 3" of gain...
Since you have been using and maintaining that 3" strip for so long, I think you have an excellent case for adverse possession. When you call his bluff, you might ask him if he knows what adverse possession is. If he dosn't, tell him to ask his lawer about it...
Good Luck.