Ray,
That only works if the exterior wall insulation is also a vapor barrier like a closed cell foam. My experience indicates that mold is rarely caused by summer driven moisture from the exterior due to dew point issues. Most it caused by poor sealing/flashing/etc and rain water entering, or leaks in pipes.
In my area, I can have a maximum temperature gradient of up to 100* in the winter (70 in the house, -30 outside). In the summer, it is about 25* (75 in the house, 100 outside) But I do not use AC, so my house temperature gets over 75 if a heat wave lasts for more than a day. It is the extreme difference in outside vs inside temperatures that drives the effect of dew point condensation when dealing with humid air. Most of the time, at least here in MI, there is a long period in the winter where that difference will be about 40* for months at a time. In the summer, a difference of 15* may last for a few weeks and the small amount of condensation that is formed will evaporate if it can escape. It is one reason, that house wrap is permeable...to allow water vapor that may be in the stud walls to exit the house.
Where I have humidity issues is the basement; in spite of having foam boards on the foundation side of the stud walls and a VP under the flooring. I use a commercial dehumidifier to address it.
In any case, what we think is immaterial when building a house or a structure that must pass inspection. Local building codes must be met. Our opinions can address how we build other structures, and to address the OP I think we both can agree he is fine with unfaced insulation for his project.