Here are a couple of thoughts:
Bury your second conduit as far as you can from the conduit with the power line. A foot of separation is good. You don't want to induce noise on the data lines in the second conduit.
Number 4 copper has a resistance of 0.29 Ohms per 1,000 feet. You will have 240 feet of copper (out and back), so that's 0.069 Ohms. If you pull 50A, your voltage drop will be 4.2V. That doesn't look like a problem to me. If you were to put in bigger breakers, you could go up to 85A to protect that conductor size (assuming it's rated for 75 degrees C), which would be almost 6V drop. That's about 5% drop, which is about as big as I would be comfortable with. Might as well put in an 85A breaker to feed the cable right now, and an 85A or smaller breaker in the subpanel. It won't change the cost enough to notice, and there is no point in handicapping yourself.
If you ever want to use 120A in the barn panel and 130A to feed it, you will have to go up to number 1 copper. That would give about a 4.6V drop at 120A, again acceptable.