A "skid steer" as its name implies, steers by skidding. Every time it "skids", it chews up some ground material. FWA tractors have front and rear - and sometimes mid - differentials, so there is alot less "skidding" to maneuver around. If you are handling logs on frozen or well-compacted gravel or pavement ground, a skid steer and a FWA tractor will be pretty much equal with respect to the chewing-up the mud. If ground is just dirt, or mud, in short order a skid-steer will make a really mucky mess out of the operating area. The FWA loader tractor will chew the dirt into mud alot less.
You mention moving logs into a mill. I presume a small mill, so you move logs for an hour and saw for ten hours. So if your moving of logs takes an hour and 20 minutes, no big deal in the grand scheme. Especially if the yard isn't a mucky mess deeper than your gum boots.
If you are working a 12 hour day with a skid-steer, you will get ~50% more work done than a similar loader lift capability sized FWA tractor. Pick up a bucket of material, move it around, and place that bucket of material accurately somewhere else, a skidsteer just plain gets it done faster. A heel-toe hydro tractor is WAY faster than a gear-grinder, but no where near as quick as the spin-on-a-dime skidsteer.
The FWA tractor is better suited to many other tasks around the farm, haying, grass-cutting, and on and on.
Think through the work you do in a day, how much you will use loader/tractor/skidsteer, and how much "loader productivity" you want to trade-off for versatility and lower ground disturbance.