Really, all you are wanting to do is put the generator in the barn and run a long extension cord from the genny to a breaker in your main panel that is interlocked, so that it cannot feed the main panel at the same time that the main power is on.
Your conduit is large enough.
You'll need a cable from the genny to an outlet (assuming you aren't hard wiring it).
An outlet.
A cable that runs from that outlet to the main panel in the house, NOT the sub panel in the garage.
A breaker in the main panel.
An interlock kit for that breaker in the main panel to select grid or genny power.
Don't try and be frugal and double purpose the ground and neutral for both functions. Why?
Because you'll confuse someone, somewhere, someday.
Don't run the wires for the genny through the sub-panel at all. Keep them seperate to avoid confusion.
Take a look at this schematic. It's probably similar to your current setup. Anyway, print it out and add in the genny circuit. Technically, I see no reason why you could not use the neutral and ground for both situations, as only the two hot legs are switched. I just would not do it because you'd have two wires running into the sub panel for the neutral and ground and two wires passing through the sub panel that do nothing. Keeping them seperate just makes more sense.
I have no idea if what you want to do would violate code or not. Running seperate would definately be up to code.