I think another thing is that years ago the manufacturers basically only made one model in a HP class. They may have had utility and row crop versions, but the driveline components were essentially the same. Look at Deere today; you can get 55-65 HP in a CUT, a 5D, 5E or a 5M, a model for every budget. The 5M would certainly match up to the old tractors but you will be paying deerely for it. The other tractors are going to be built lighter, smaller displacement, high rpm engines, etc. A lot of the older tractors mentioned could be had with cast center wheels, large diameter tires, etc. Even the Ford 4000 mentioned could have been ordered with 38" rubber. Carter's old JD B probably has some tall skinny wheels on it. To get that kind of stuff today Deere would have you move up to one of the premium lines.
Part of it is what deere5105 says; large farms today aren't buying that many tractors in the sub-100 HP range, so to keep the price down for the smaller operators they come out with the lighter economy models. But when you choose one of those you give up a lot of the user friendly features found in the premium lines. As far a being more nimble, I'm not sure. If you have to move up to 4wd to get the job done a lot of the nimbleness can be lost.