It would be good to know how hot the hydraulic oil is when you are having this problem. The reservoir capacity is small considering the flow rate capacity. You may have cooling issues or excessive wear on the pump but there are a few things to consider before pulling the pump. I think that it isn't as likely to be a restriction in the lines or filters as you say it works fine when cool.
The transmission pump (gear type) is pumping IIRC whenever the engine is running and the clutch is engaged. Don't know how it bypasses when the main pump is out of stroke (no hydraulic functions being operated) and full pressure is reached. Will have to check the drawings to see.
If it is similar to the 4020 the priority valve shifts at pressures below 1900 psi to steering and brakes only, so does the rockshaft work at startup then it fails shortly before the steering fails?
Do you have a pressure guage to check the system pressures cold and hot?
I would also add there is a filter above the drain plug on the bottom of the pump that is about the size of a shotgun shell that I had fail on my 4020, for a 4640 this is part # R27173 on the stroke control assembly. This is easy to check and get to from the bottom of the tractor, remove the drain plug assembly which has a small rod with this filter on it. This small filter screen if plugged up will shut down the pump. If you look at the stroke control in Deere parts it is marked as #17.