If you wish to discuss any aspect of the Vanguard, please do not call it a "motor"..... call it an "engine" ONLY. The term "motor" is reserved for the hydraulic motor bolted to the trans-axle that propels the tractor. When you mix the two terms, it becomes confusing and that makes it difficult to help you. OK?
The fact that oil was overflowing, indicates a foaming condition that is caused by air being sucked into the INLET side of the hydraulic pump. That pump is moving the oil but it is moving foamy oil, not liquid oil. When air is entrapped in the oil, then the system cannot build pressure to perform any real work because air can be compressed.........pure liquid cannot be compressed.
As I stated earlier, if you are using those worm-screw gear clamps on any of your hoses in the hydraulic system, then that is a mistake that needs to be corrected. What you need are the flat band clamps such as the ones you see here.
McMaster-Carr
Also.... make sure that your pump inlet fitting is sealing properly. Some OEM pumps used a moulded steel elbow that was pressed into the pump with an O-ring on it to seal between the fitting and the pump body. If that O-ring is missing or damaged, then air will get sucked past it. Aftermarket pumps use an O-ring Boss fitting (ORB). The inlet is threaded and the inlet fitting screws into it. In order to seal that connection, there is a large nut on the inlet fitting that forces a large washer against the pump housing and squeezes down on an O-ring to make the seal. If the O-ring is damaged or missing, then air will get sucked into the pump.
Focus on the J-hose, the clamps that secure that hose and the fittings that go into the pump body.