Did the chips get inside and do any damage to the ball bearings?
Hard to say what could have caused the pieces to chip off in the first place.
The true fix would be to replace the shaft, seal, and bearing.
Emory cloth would take forever but maybe a Dremel tool or die grinder could smooth it out enough so it won't chew up a new seal.