A 4020 is a big tractor, but it's not huge. Designed for ag purposes, yes, but it was designed for ag purposes 60 years ago, which were a lot different than today's tasks. Frankly a 4020 was probably designed to be quite versatile, being a good choice for plowing, planting, cultivating, cutting hay, baling, filling silos, you name it. But they were also small enough to do chores with, feeding animals, mowing fields and ditches, plowing snow, grading the driveway, stuff like that. Also, they were designed to put several thousands of hours on them without trouble. My friend grew up on a used 2520 with a gas motor that they still have on the farm. It's got over 8000 hours.
A couple things I'd think about: from what I know, the price of a 4020 changes dramatically if it's gas or diesel. The diesel would be much more desirable, and use a lot less fuel (probably 4-5 gph instead of 7-9). Also, whether it is the newer "side-console" design will impact the price. I haven't spent much time on these machines, but my friend who's in the know says that the side console is where it's at. The earlier models had the three point controls on the dashboard, which makes looking behind you while making adjustments (something I do every day) much more difficult. From what I've seen, the gas models usually go for $5-8k, diesels from $7-15k.
Beyond that, I think a big JD like that might be an okay choice. Personally, I'd probably skip the loader, unless you could quickly and easily remove it. A 4020 is big enough as is, adding a big implement to the front will make it pretty awkward. Plan on working on it, certainly, but parts are available, and the 4020 in particular is one of the most popular tractors ever made, so I'd imagine there's lots of know-how to be tapped both online and in rural America.
One last thought, my MX 4700 is a great tractor. For loader work and all the stuff I encounter day to day, I'll pick it over a piece of old iron hands down. But for pulling a big mower on a big field, a big, long, wide tractor like that 4020 would be a much more comfortable ride. Someday I'd love to have one around for pulling heavy wagons, big logs, and maybe mowing, if I could afford an appropriate mower.
Good luck with your decision, and let us know what you end up with.