Build the barn first and/or a garage. Your young even if you have young kids, you can live in an open floor plan for a while. It is nice to be able to be right on top of things when they are building the house. That was our original plan, however wife got the house first. Shop didn't happen for 10 years latter. Having a shop available will save you money with the ability to fix your own equipment. Think of it as camping in your garage for a year. Looking back on it should have stuck with original plan. At the time we sold the house paid cash for the land, moved into a duplex, minute they set the house, chances are going to have the two payment thing. That eats up a lot of cash right when you need it most. No matter how you slice it, if you taking a loan out your going to have to pay for it eventually, the key is keeping it so you can make payments on one income.
You have, road, well, septic, power, gas?, foundation, then set the house. The big ones are the house and the land. The quicker you get out of current mortgage, that is money in the bank. Once you have the well, septic and power you can live in a trailer for that matter. Just something to kick around.
We set the house in February, crane set, that way you can confirm a schedule, ground frozen. You can swipe snow off. Even though money is cheap still have to pay it off. We built when I was 35 taken us twenty years to get to the point where we have disposable income, with a lot of stress along the way.