The problem, as I see it, and as a gravel hauler explained it to me, is that too many fines retains the water and creates puddles. Now if you can get enough crown and slope to eliminate the puddles you won't have any problems. But puddles at the surface will cause problems as described below.
When you get puddles (which almost always happens due to compaction in the wheel travel areas) the problems start. What happens is that the vehicle wheels splash through the puddles, and throw fines and small material out of the area, and so the puddle gets deeper. The more the puddle is driven through, the deeper it becomes, until you have a pothole, due to throwing additional material out of the hole each time a wheel runs through it.
Two solutions.
1. Drive very slowly through the puddles (yeah right, and try to get the neighbor to slow down!)
2. Gravel hauler says to use larger gravel (like #53 w/o fines) on top of the base of finer material. This way any water that is retained by the fines is below the driving surface. Since the wheels don't splash through the water no potholes develope. This isn't very good for snow removal, as the larger, looser gravel on top is easy to push off into places you don't want. But it works well during the summer, especially if it is rainy. And it helps to cut down on the dust as the fines are below the primary driving surface and not stirred up quite as easily.