Thanks for the additional reply's everyone. I haven't checked in on this thread for a bit as things got busy at the ranch leading up to Thanks Giving.
To address some of the points made.
Yes, there is no way to block off this guy without blocking off the other people who do have easement rights. I wish it were that simple!
My driveway is not a public road, the county even has a sign stating that it's a private drive. I have that part of my property surveyed and I've also talked at length with the county surveyor about who has what easement rights to said drive.
Re-guarding easements... they were on the top of my list of things I didn't want when buying land. It was pretty much non negotiable.... but as they say, never say never. The wife found a place that was simply too good to pass up. A once in a life time opportunity. I still can't fathom how lucky we were to get this place! The only real downside was the easement. But the easement comes with a long list of restrictions for the three neighbors down stream from us. For example they can't run a business, they can't subdivide their land, they can't park on the easement, they can't do anything on their property that's considered a nuisance etc. The drive is for ingress and egress only. Even though we are part of the same subdivision, our place is not subject to any restrictions (unusual I know). Also due to the way the covenants are written, we have the controlling vote over maintenance. So anyway, as far as easements go, we have a pretty good one, that doesn't impact us much provided people follow the subdivision rules (which they are legally obligated to do).
However the "secondary access" guy is not part of the subdivision so none of this applies to him. Anyway, I haven't seen him in my cameras. I'll keep an eye on it. Our plans have always been to put up a nice arbor and a electronic gate at the entrance to our property.
I obviously won't be giving him access, but one of the other neighbors might give him the pin number. I honestly don't know how they feel about him using their private property for his secondary access. So for now I think we'll just wait until we can block his access and see what happens from there.
I appreciate the legal advice and council regarding whether it's even worth it or not. He's an old guy so giving him permission might indeed be the way to go. It doesn't sit right, and we do have the funds to hire lawyers, but honestly that's not my style unless push comes to shove.
Once again, thanks for all the help and wonderful advice.