I'm not familiar with what a local town board does, or what a farmers co-op is, but I might be tempted to go to one of their meetings and ask them for advice in how to deal with your situation without naming names. Bring pictures of what's happening, show the survey, and ask what would they do if this was happening to them on their land?
...... He is probably just thinking he is not hurting anything.
.
Just a "devil's advocate" question - what IS he hurting? Is he damaging your property in any way or are his actions denying your use of your property in any way?. Yeah, I'd be pissed too, but then - why?
For liability reasons I would not want him to illegally use my property if it were me. Also the adverse possession mentioned.