From a Civil Engineer who JUST installed a 350 sf brick patio:
To serve as an adquate base for a patio, you must to rent a 'jumping jack' amplitude-based compactor, compact the area of interest before you put anything on it (as it sits today). Then you can bring your additional soil in with 6" lifts and compact each layer as you bring it up. Only with amplitude-based compaction can you 'tear' the soil and get effective hardening of the subgrade. Keep going until you can't drive a stake or landscaping nail into the ground. You'll know.
As others have said, organics in the topsoil doesn't work...it never 'tightens'. Free junky junk will always turn to soup no matter what you do to it, unless you get another truckload of LIME to go along with your 2 truckloads of junk. And you don't want that. It is expensive and very, very messy.
Simply placing the soil in one spot and litting it "settle" will not effectively compress the entire area or, most importantly, the subgrade. And it won't happen consistently, so you'll end up with sinkholes. And using a compactor after you install an 18" deep lift won't allow you to compact the deeper section of soil, leading to the same problems. The folks telling you this probably use the term "surcharging" the base, but you need a HUGE MOUNTAIN of soil to effectively surcharge and, most importantly, you need to install a french drainage system since the only way to 'tighten' the in-situ material is to push all the water out of it. You're not doing either (certainly not the first and probably not the second), so making a big pile is pointless.
Using the front tires of the tractor with a loaded bucket is OK for really (really, really, really) light duty stuff or things you don't really care about, but it is insufficient for anything that matters.
Before you install the stone, be sure to lay a WOVEN geotextile fabric between the stone and soil. In case you have any weak spots in the soil, the geotextile fabric will add another structural element to "hide" minor imperfections. Don't use NON-WOVEN fabric as it does not have a structural component/benefit and simply acts as a media seperator. The fabric is cheap. Hold it in place with sod staples.
When you add you gravel (proper base for any patio, be it concrete or brick...4" to 6" in depth, depending), be sure to properly compact it in lifts as well, but use a FREQUENCY-based compactor, also called a 'vibration plate' compactor. Graded stone (crusher run with fines...never use evenly graded stone) does not respond to the jumping jack...graded stone relies on the vibration waves to fill in the gaps in the stone with fines to create a hard base. Work from the perimeter inward to force the stone "into" itself, then make multiple passes in different directions.
And when you add sand, never use "concrete dust". It is junk. It holds too much moisture and is inappropriate for a structural application no matter what ANYONE tells you. Trust me. I know. Use WASHED SAND. It is more expensive, but it is the right material to use.
Happy to discuss further if you have any specific questions. Feel free to PM me.