You haven't even scratched the surface of expensive. By the time you install doors, windows, interior partitions, wall and ceiling finish, floor finish, wiring, plumbing and HVAC, you will find that building a pole house didn't save you much money, and cost you resale value.
It is true that the building "shell" is way less than 50% of the total cost.
In the area we live cost per sqft of standard house was about $110/sqft and up in 2004. We built our house for about 85-95 sqft. We splurged on silestone counter tops, marble tiled bathrooms, maple or ceramic tile floors, upper end appliances, geothermal floor heating and 5 car garage. The cost includes about $5000 for quarter mile long driveway, 11500 for pond and $12000 septic. Our previous house was stick built nevertheless it was different than houses around. We were also warned about low resell value. When we decided to built on our land we put our house on the market and it sold for asking price two weeks later forcing us to live all summer, while finishing our current house, in the garage. Just to get some perspective we looked at houses for sale in the trendy posh area of the city (150 to 180 per sqft I am guessing). They were "loaded with features" such as breakfast room, formal dining, den etc. (Those are code names for "useless" IMHO). Our experience teaches us that life and our parties happen in the kitchen. Therefore none of houses we built was loaded by "features" but the focal point of the house was big kitchen opened to a big living room. Before we built our first house we were invited to many parties with friends and it always happened in the kitchen. Based on the observation our houses had "live and party in kitchen".
My conclusion is that there is a buyer for every house. People buy house for different reasons. Some like "features" and some want practical livable house.
It doesn't matter if the house looks like a barn or McMansion.
Another note: When we were building our house the "shell" with all perimeter windows and doors fully insulated, reinforced concrete slab was $35/sqft.
We are building another Morton building with similar parameters but the cost is $25/sqft.
The advantage of pole barn is that you can do a lot of work by yourself. At least in the area we live. We saved estimated 60 to 80K by our own work. My wife pulled all the wires, I did plumbing, we installed HVAC, laid floors, installed all internal insulation etc together.