The bottom of the holes were virgin undisturbed soil. Gravel was placed in the hole (6") to keep water away from the base of the posts. Once the post was placed, it was dropped up and down repeatedly to compact the gravel. So the post was setting on a solid footing.
Clay soil can support about 2000psf, pure gravel is around 5000psf. So unless the concrete collar was engineered to become an integral part of the post, ignore it and figure out how much load the post can support directly on the gravel. Either the gravel or the soil will shear first, need to figure out which. If it's 6x6, each post is 1/4 square foot. The gravel supports 5000psf so that post will support 1/4x5000= 1250 pounds into the gravel. Loading spreads about 60degrees from horizontal in gravel so if there's 6" of gravel, the load is projected out to the side about:
6*tan30= 3.46 inches. Post is square but assume it's round 6" diameter to make things easier, so the increase in area is about a factor of 10, which means the 1/4 square foot post loading is spread over about 10/4 sf of soil, call it 2.5sf. So that's 1250lb/2.5sf= ~500 psf, which is well below the capacity of most soils. So the limiting loading is into the gravel and you can have about 1250 pounds on each post.
Not sure what a building weighs but it's not that hard to add it all up and divide by the number of posts. If that number is less than 1500 it should be OK, if it's greater, probably not so good.
For comparison, a post sitting on a 16" round concrete pad has area:
3.14x8**2/144 = 1.4sf
If it's on 2000psf soil it can support:
1.4sf*2000psf = 2800 pounds
Which is about two and a half times putting it on gravel. That means you might have to have 2.5 times as many posts in a pole barn set on gravel to support the same weight as if you put it on 16" concrete pads. That assumes it's all well drained, add water and all bets are off.