What does or will your lot look like? Will it be mostly open, or will there be a ton of trees and things to maneuver around? If it's mostly open, or even partly open, a compact or subcompact tractor will be the best bet, because you can have a loader, run a tiller, aerator, etc conveniently.
Of course you figure with the dump trailer and Jeep, you'll still have to find a way to load it in the first place, unless you're just going to make a bunch of trips back and forth to the place they're loading the dirt. After loading the trailer up by hand a time or two, you'll be ready to get a tractor/loader.
Now if after you get the majority of the big work done, you could probably sell the tractor for a good percentage of what you paid for it, and put that towards a quality ZTR.