Not sure if your quick-connects are the same as mine, but if they are, once you slide the female fitting collar back all the way and push it onto the male fitting part way (but not yet latched on), you no longer need to hold the collar back - the latching balls inside the female fitting will prevent the collar from sliding back down until the fittings are fully seated together, at which point the collar should snap itself back down. I didn't know this at first, and was trying to simultaneously pull the collar back and push the two fittings together (sorta difficult in my confined space). It helps a lot to only have to worry about pushing ONE direction at a time!
What everyone else said about any residual pressure is very true too. In some cases, if the loader is left off and in a sunny spot, the sun heat can cause enough pressure buildup to make connection difficult. Some have found it a lot easier to simply crack one of the threaded fittings in each circuit open slightly to bleed out excessive pressure.