Here are a couple photos from additions I have added onto my pole barn over the years. This older addition had about a 3' drop to the rear, which I handled with an extra skirt board (marine lumber, will outlast me) and a crude log retaining wall:
At the time I made a mental plan to come back and make a nicer retaining wall or just dump gravel there and slope it, but that was about 5 years ago and the clock is still ticking on my plans. Surprisingly, the logs are still solid too, but that won't last forever. I'd probably use marine lumber for the permanant retaining wall (they make a tongue and groove material for bulkheads that is real nice). But the prices will need to drop/stabilize before I spend that kind of money.
Here's another addition where I used an extra skirt (ground contact 2x10) and then let #57 gravel spill out from under the skirt boards until I had a nice sloped berm. The berm was topped with crusher run and then wood chips, and has held up well. That was probably about a 4' drop at the low corner (closest to the camera).
I put a dead-main in the middle of the rear skirt extension (you can see the bolts and plates), just in case any pressure was put there by the gravel. So far the skirt has not budged after a few years of parking cars in there, so either it wasn't a problem or the dead-man is doing it's job. If you have sufficient posts that can handle the retaining load, then a dead-main isn't needed. For instance, on the eave walls I have a lot of posts to support the header and roof load, and did not put dead mans there.
I think if this was planned properly with large long posts, you could incorporate a retaining wall (full or partial) into the main poles of the barn. They'd have to be long enough to frame the wall and go down below your frost line to a footing, so that could be the issue -- those might be very long posts when all that is taken into account ($$$ and or availability issues).
That assumes you are wed to the pole barn method. I happen to like it myself for many reasons. But it works best and is the most econimically viable when you can build on flat or mildly sloped ground and if you are OK with a gravel floor. Once you pour a concrete floor in a pole barn, you may have been better off building with standard stick-framing on a slab, or stick framing on a concrete block foundation.
BTW, you can do a concrete block foundation that will handle a 6' drop and then backfill that with gravel. It just needs to be engineered properly. Don't know the cost or feasibility of that myself. I just know that the garage on our main house has a block wall and sloped about 4' front to back. The back of the foundation has 4' of gravel and concrete debris to fill it out and support the slab.