Update:
Couldn't sleep anyways, so I got up early this morning and started researching. I bought the pdf 5155 service manual from Bill's Tractor. Got it downloaded, turns out I was sent the manual for the 4500 series by accident, I am hoping. Regardless, nothing useful regarding shift linkage for the 5100 series.
So, I went outside and started to really inspect the linkage, which I should have done sooner. The shuttle shift cable connects to the shift lever on the side of the transmission. It is located beneath the floor boards between the fuel tank and the transmission as shown below. The cable shifts the lever towards the rear of the tractor for reverse, or towards the front for forward. There is a small range of adjustment in the clevis at the end of the shift cable.
View attachment 744941
I suspected the shifter was not shifting the transmission completely into reverse, causing it to pop out and grind under load. I backed off the retaining nut and removed the connecting pin, which is held in place with an attached tab that snaps around the shaft of the clevis. I rotated the clevis on the threaded rod two full turns to lengthen the throw of the linkage, thus shifting the transmission "further" into reverse. This adjustment was small enough to not affect the Forward shifting throw.
Replaced the connecting pin and tightened the retaining nut, then did some testing. Made around 100 forward/reverse shifts in the yard without it popping out of gear. Tried every gear with different amounts of throttle to simulate different loads. I put it in 4wd and stopped the tractor facing downhill on a slope. Then I backed up the slope using a variety of gears and throttle, including some quick clutch releases in high-range/1st to really put the shifting mechanism under load. Never popped out.
I'll be posting if it happens again, but knock on wood adjusting the shift linkage seems to have cured the problem.