When doing repairs on your gravel driveway - potholes, washboard, ruts, washouts - it is important to first thoroughly scarify the area being repaired. Without scarification - there will be two differing materials at the repaired spot. Example - the original pothole and then, material drug in to cover the pothole. These two "soil stratas" will not mix, intertie or bond together. The initial pothole area will need scarification to get the overlaying material to bond and make a permanent repair.
You can drag material and cover the pothole without scarification but slowly and surely the original pothole will return. Essentially - there is nothing tying the cover material to the base material where the pothole is.
Remember how you repaired an old inner tube. You scuffed the area around the hole so the patch would hold. That is the EXACT same thing you are doing when you scarify the area around a pothole, etc, etc.