It has to be a problem with your joystick valve. When you raise the loader, fluid flows in the same lines as when you lower it, just in the opposite direction. That means your valve must not be opening properly when you move it to lower the bucket. I'd bet that if you reverse the hoses at the quick connect, you will also change the problem from lowering to raising. That would tell you it's the valve for sure.