I set many posts as well as plant trees with a post hole digger (3 point). We have sandy loam here in S.E. Michigan and I never put water in the hole as the sand is abrasive enough when dry.
What I do is run the PHD to just about the bottom of the hole, disengage the PTO and pull the PHD from the hole. The sand will stay on the flites. The I move away from the hole, engage the PTO and fling the sand off. Back up to the hole again and run in the PHD to the required depth, shut off the PTO and pull the PHD. That way, very little material falls back into the hole and you are good to go. I have done this literally hundreds of times and it always works.