I've had fewer problems each year since I bought a couple hundred acres almost 10 years ago. At first I had to deal with "I've always ridden here and hunted here and I'm not gonna stop now" attitudes. I'm not proud of it, but after one hunter made a not so subtle threat with his shotgun, I held him at gunpoint until the authorities arrived. I'm not going to get shot over someone trespassing or trashing my property, but when someone threatens to use a firearm on me I will draw mine immediately. Apparently he didn't want to get shot for illegally hunting either. No matter, I just don't want to get into that sort of situation again.
I then simply used my excavator to make the 'established' trails (not established by me) impassable and dug a 3' deep ditch with all the spoil on the back side down the road side of my property. Even an ATV can't get across that unless they come prepared with a backhoe first. Anyway, my biggest issue now is hunters and their messes. Each fall and winter I probably collect 4-6 deer stands off my property and get to clean up all their empty beer cans and other trash. I'm hoping they'll get tired of losing deer stands and just move along elsewhere. I had setup two game cameras and both ended up stolen, so that didn't work out so well for me even though I thought I had them hidden well.
Short of having someone patroling the place for you, I don't see any way to keep people off remote property. I just make it as inconvenient as possible and spread word that I will prosecute. I seem to have fewer problems each year. Hopefully that trend continues.