On my PTO pump driven mower, I wanted the mower and the tractor engine to turn at the same RPM as when the mower was on the TPH. That required that the displacement per revolution of the PTO pump and the mower hydraulic motor be the same; actually, the pump should be 5-10% larger than the motor to make up for pump and motor leakage (the leakage is not that much when pump and motor are new, but it increases as they age).
So if the displacements are to be the same (or nearly the same), the next step is to determine the displacement needed to transmit the power available. I am working with rough figures and doing the math in my head, so the following figures are only approximate; precise formulae are available in handbooks or on line. A 6CI/REV gear pump at say 2,000PSI takes about 19HP to drive at 540RPM and produces about 13GPM , which in turn results in about 16HP being applied to the input shaft of the mower gearbox.
So, if your tractor produces 19PTO HP at 540PTO RPM, a 6CI pump will handle all the power that can be produced. A larger pump and motor will not provide any more power to the mower; and a smaller pump and motor will not transmit the full PTO power. And a larger pump and motor will increase the GPM that flows through the system, which in turn increases the requirements for hose and fitting sizes, quick disconnect sizes, and reservoir size. So nothing is to be gained by increasing the pump and motor size.
If your tractor produces more power than required to drive the mower, you can achieve a relative RPM reduction in the tractor engine by increasing the size of the pump relative to the hydraulic motor. In the above example, if your tractor produces say 38PTO HP at 540PTO RPM, but you only need the same 16HP to the mower input shaft, you could increase the pump size to 12CI/REV, run the engine at 1/2 standard speed, get 19HP at the PTO, and still produce the same 13GPM needed to drive the mower (I know that HP is not necessarily linearly related to engine RPM, I am just assuming that for simplicity).