So I just built a pad for my new 30x60 pole barn. My pad is all fill since I find it pretty silly to dig a hole and put a building, especially a pole barn, into the hole. My building is on a slight slope just like the OP's and the fill "wedge" goes from 0" to about 30". A very important thing to do in this situation is to build the pad wider than your building, meaning, keep a level area outside of the poles for a good distance, in my case I went 10 feet beyond the building. This is to get good compaction aroudn the building since near the edge of the fill and certainly on the fill slope you won't get good compaction and the poles are using that material for lateral strength.
First thing I did was strip the sod. Can't tell if you did that. Then get inorganic fill dirt delivered or dug out and ready.
Then go rent a 3 ton vibratory roller. These things are great and only about 4 feet wide. With a roller like this you only need to grade each lift close and then the roller makes it all look pretty.
I then started spreading fill. One scoop at a time. Go easy, you only want about 8" of fill which will compact to less than 6". Then blade it smoothish with the boxscraper. Then compact it with the roller in both directions at least a couple of times. Moisture during compaction is important. Too wet is better than too dry. Mine was too wet at times which makes the roller do some funny stuff.
There is no difference between properly compacted fill and native ground. You can make fill dirt hard but you must do it properly. You can even have a company come and and test your soil for compaction if the risk warrants it.
My poles will be drilled deep. Deep enough to go through the fill and onto native ground just like the OP's. This means that the only risk with fill is the differential settlement breaking your slab. Is that a big deal? Really? Come on now, all concrete cracks. Good steel reinforcement will help prevent the cracks from seperating.
You can always pour more concrete on top of the settled concrete. You can always break it out and replace it. You will have to decide how many inches of settlement is too much.
To the OP: You've probably already built your building. Removal and replacement isn't an attractive option. If you can wait, fine, I don't think it will get you much if the roof is on. Your problem isn't settlement, it's differential settlement so expect to see the slab roll down where the fill is deepest. The back of the slab will settle more than the front. The building is bearing on the undisturbed earth so the building is fine. Worst case, you repour the slab if/when it gets bad enough. May not even need to remove the old slab, just pour a new level slab on top of the settled one.
Yes, you should have compacted it better. Too late for that now.