Hi Parry,
You're probably right about it being loose. That is a sure way to have that tapered portion on the threaded shaft not seat into the tapered hole snugly. However, do you see the cotter pin in the shaft?
It aligns the castle nut and determines where the nut can be tightened to, if you want to use that cotter pin. The other thing is that the taper part on the shaft is shorter than the thickness of the casting hole. So if the shaft is not super tight in the hole, you're going to get that egg shape in the hole, and part of the threads will wear as it wiggles around in the hole.
Perhaps a cure to that would be NOT to use the cotter pin to keep the nut on, but rather a double nut lock. That way the shaft can be held in super tight against the taper without trying to line up the cotter pin hole with the castle nut. I'll bet if you check yours, and your are using the cotter pin, yours might be just like mine? Hopefully not, but it's possible.