Check your deed and verify the location and width of the driveway. Is there a road maintenance agreement? Check the court house records for this information. I have seen situations where people are part of an HOA with a RMA but did not know it. From the history of the place, I doubt there is an agreement but it would not hurt to check.
Once you know the legal description, you need to make sure that it matches what is on the ground, which could mean a survey. If you can find the pins, and they match the paperwork, you might be able to skip the survey. If it was me, I would get a survey especially since you are putting up a fence. The survey will nail down the property lines and I have seen people surprised at the location of the lines.
I would also talk to a lawyer, no, not to sue the guy, but to find out what your state law says about this situation. You might not be liable at all for the driveway but maybe you are liable. Since the driveway is NOT for your use or benefit, I don't see why you would have to pay to maintain the driveway. If the use of the driveway is doing damage to your property, who is liable? Over the years we have talked with quite a few lawyers for advice and we have not been billed once even though we asked and expected to pay.
Before I start down the path similar to what you are having to take, I like to have my ducks in a row. If that means paying a bit here and there so be it. I want to be R I G H T and it is cheaper in the long run. If you can say to a judge, jury, and/or neighbors that I checked the court house records, got a survey and talked to a lawyer, you have pretty much done all that can be done.
Having said that, go read the huge, long running, thread by MDBARB about his easment problem,
http://www.tractorbynet.com/forums/rural-living/14251-help-i-need-legal-advice.html. Actually, the thread is so long, just read the first few pages and the last few pages. Bottom line he lost and I believed his side of the story. He lost a huge some of money.
I do have a concern about the fence. With his heavy equipment, and the fact he hit his sisters car, I think he will take out your fence and then you are into a war. If you put up a fence I would have large, heavy rocks placed to protect the fence. If large rocks are not available, you really need to put in heavy posts outside the fence line to protect the fence. Put in something you do not care to loose or that is hard to replace.
Its a shame that we have to deal with people like this but it is life.
Good Luck,
Dan