Road base gravel needs several things to work. First it has to be on a solid surface with good drainage on both sides that quickly takes the water away from the road. If you have good drainage and since the road has been there awhile, you should have good compaction. The next thing you need is a proper mixture of big rocks down to fines. The rock has to have jagged edges and should be no bigger then a couple of inches. The road base rock needs to be at least four inches thick, with 6 inches being even better. This is really important because if it's not thick enough, it will not lock together. Once locked together, it becomes a solid surface that water will not penetrate.
Potholes are caused by one of two reasons. First is that the soil isn't compacted. This can be from just not getting compacted when building the road, or something in the soil broke down over time, such as organic material like top soil. The other, and more likely reason for potholes is that the rock wasn't thick enough to lock together and water got through it when it rained. Then every time you drove over that spot, the fines washed out and you where left with just larger rock. Then over time, that rock moved away from that area when the tires hit it.
One of the biggest mistakes anybody can make is to smooth or grade a road that has been compacted. This breaks it up and allows moisture to get into it and wash out the fines. The more you do this, the worse your road will be. Once a road is properly built, you should never grade it again.
Without seeing pictures of your road or knowing all the details, it is my guess that you do not have enough material on your road and the only way to get it right is to buy more road base and get it to the proper thickness, then compact it. I would tear up your existing rock and mix it in with new, then spread, grade and compact it all together. Do it right and it will last a decade or more. I'm at ten years on mine without needing any additional rock and I wont be surprised if I get another ten years out of it.
Eddie