In my opinion (for what that's worth) you didn't get the nut near tight enough. Okay, that said, here's what I do.
When I did the first one (many years ago) I made a socket to fit the nut. I took the nut off with a hammer and punch, basically beating the snot out of it and rendering it pretty much unusable. I feel it was worth my time and effort because if you check the part number applications you find the same nut fits a multitude of different models, even far bigger tractors. It's rather crude but effective. It's basically a short length of pipe, probably about 3" diameter or so, whatever size it took that it just fits over the stake rim on the nut. Four pieces of 1/2" keystock strategically welded on so each fits into one of the four notches on the outside of the nut. Then a 3/4 drive socket welded into the other end of the pipe, and that's my tool.
Before I even begin to assemble anything I trial fit the nut onto the hub. This is provided I'm considering reusing it. What I've found is the flange that's staked into the notches in the spindle is usually so messed up after removal that it takes considerable effort to screw the nut back on. This can affect what it takes effort wise to tighten the nut, and can make the nut "feel" tighter than it is. If the nut won't thread on past the distorted flange with just minor effort I will replace it. If it's been off before and has multiple dents replace it for sure. I take a thread file and dress the threads on the spindle where the stake flange has usually done some damage on the way off. My goal is to thread the nut on by hand (or at least with reasonable effort) until it's all that way onto the threads. It doesn't always happen that way, but it's what I hope to see.
In my opinion, you can't get the nut too tight when assembling. The bearing preload is self determined by the machined surfaces and the spacer between the bearing cones. The nut has nothing to do with that. I put the new seal in the hub, put the inner bearing on until it bottoms out. slide the hub and seal on by hand as far as I can get it to go. Put the outer bearing and ring gear flange on, slowly driving it with a hammer and punch working my way around. All the while on working the hub by hand if possible. Maybe a few well placed smacks with a four pound hammer to help it on, trying not to let it cock sideways and distort the seal. Then the nut goes on. I don't know what the proper torque spec is, and have no way to measure it anyway. I put the nut on and tighten it with a 3/4 ratchet and a three foot cheater pipe. I get is as tight as it will go with all my weight on the end of the pipe. As I do that I frequently turn the hub by hand to "feel" the resistance offered by the seal and look for any increase in the drag as the nut gets to full torque. If all is well, the nut gets to where it can no longer go any more and the rolling torque of the hub doesn't change. At that point I feel I've got it done. Stake the flange, put the planetary hub on and add oil.
