You won't be haying long with a JD24T or JD410. Both are nice machines, but not for high volume work. I might be inclined to upgrade balers before I upgraded tractors. A lot less capital up front and you won't be fighting baler problems (I remember those days). I would keep both the 24T and 410 as back-ups, because break-downs do happen. Keep in mind that a 4040 with 3500 hours is just getting broken in, unless it has not been well maintained. If you said 10,000 hours, then you are begging for trouble.
I love my JD5420 especially for haying. I wouldn't go any bigger for my baling tractor. The only reason I would want a bigger tractor is for running a larger mower conditioner, but in my smaller odd-shaped fields, I do not pick up a lot of efficiency going much bigger.
To answer your actual question, both tractors will do you fine.