I agree with all the above. For standard building, it's a lot better to put poles and slabs on soil that has been sifted, amended, and then compacted in six inch lifts. But there are other ways that can work if you will think outside the box and take some risk. Judging from that photo of your site you don't mind the risks. There's trash, rocks, and what looks like old concrete in your site prep. You wouldn't have those unless you were willing to gamble. The trick is to do it differently...
25 years ago we built a small heavy 2-story 20x50 barn on uncompacted fill that was way worse than yours.
You do have to drill down to form pockets for your piers in undisturbed soil. We found a a local guy who drills shallow wells to do our post holes. He had a nice drill rig on a flatbed truck with hydraulic feet and no problem drilling down as deep as wanted. The well rig bit was able to slice right through some rocks too. We went down ten or 12 feet with pretty big diameter holes...I forget just how big, maybe 12 to 16" ... anyway big enough so that we could pour concrete around the creosoted poles once we got them in and positioned.
Same for slabs. You can't put down a standard do-it-yourself home builder slab on uncompacted ground and not expect it to heave and crack.... but if you are willing to compact the top foot or two of soil you can put a slightly more sophisticated slab called a floating or self-supporting type right there in your barn. Find a local civil engineer who has has designed piers and slabs for sandy areas. They know how it's done.
What you will get from the PE will be a drawing of a slab with lots of rebar, and might be thicker than most slabs especially if it is designed with a waffle pattern bottom. It might or might not connect to the piers. But those are just details.
I put a couple of slabs like that in our pole barn some 20 years later when I turned part of the barn into a machine shop & wanted a concrete floor - and they are doing just fine. No cracks, no shifting.
rScotty