Out here on the other coast the rules about livestock and owner responsibility are pretty clear. The owner off the animals is responsible for keeping them on his (or hers) property, a neighbor can lawfully shoot and keep a stray animal on his property. Mind you it doesn't make for particularly good neighborly relations to do so, I have been involved in more than a couple of stray critter round ups over the year's. I also happen to like my neighbors and they would do the same for me.
Point is, if he has horses and you don't it is his responsibility to see to their confinement and welfare, not yours. I would defiantly put up a fence, if it is not sufficient to keep his horses off your property so be it, he is responsible for any damage to your property (including your fence). While the cost of the fence may be high it will probably not be as high as getting him back on his side of the property line if he is allowed to encroach.
As to his nervous horses, that to is his problem and not yours, any lawful activities you engage in on your property that disturb them are not legally your problem. If he needs a buffer it's on him to provide it.
Get this situation under control sooner than later,this guy does not sound like a good neighbor!