Pot holes form when water is sitting on a gravel roadway, and in wet weather, traffic pounds into these pockets of standing water. That tells me you have more issues than just a sandy gravel base, it tells me you lack a crown/inslope/outslope to your roadway. That has to be established first, just putting down hot-top grindings is not going to be the silver bullet, and paving it means the pavement will break up in very short order because the base is messed up.
Always start with drainage. Do culverts need to be placed in the worst locations to keep the water moving from one side of the road to the other? Ditches made? Ditches cleaned out?
Once that is addressed it may be time to think about geotextile fabric. I hate the stuff, but I do not have sand so I can fortify my heavy haul roads with rock. With sand geotextile fabric will do wonders because it will keep your gravel from mixing into your sand. Again, you probably will not need to use it along the entire roadway, but just in the problem areas.
Once those problems areas have been dealt with, it is time to take a long hard look at the roadway. Most likely, years of maintenance have meant the crown of the road has been taken out, or there is crown where inslope and outslope should be and it is not there. There is no one size fits all for road surfaces, they can change from crowned, to inslope, to crowned, and then outsloped and back, or any combination thereof. Just be wary of outsloped roads if you live where there is snow and ice a lot, and the road is 4 season. Still, getting that water to shed off the road surface will go a long ways to preventing potholes in the first place.
Written down, this probably seems overwhelming, but I am sure you only have a few problem areas. Most of the time pot holes come about the entire length of a road because people have a box scraper and drag the living crap out of their roadway. Sure it fills in the pot holes, but takes the crown out of the road too, making pot holes form quicker, so they scrape it again, and soon they have pot holes all the time. It is a vicious cycle. Getting the water off the roadway as soon as possible is what stops that, so crown or slope is critical in a road.
Here is a after and before picture of one of my Heavy Haul Roads, showing proper crown. This was done with a 2500 Kubota Tractor and a dumptrailer, so it does not take a lot of specialized equipment.