When you get in to the seal, try to remove it without damaging the sealing lip.
Then you can check whether it was worn or damaged, check also for wear from the lip on the shaft, if the shaft is worn see whether you can get the seal in or out a bit next time or fit a speedie sleeve.
Check also for slack in the bearing or bush before reassembling.
Allenr