I guess I will be in the minority. I don't think a 4320 is necessarily too small for your tasks. I do need to know how many acres of pasture you're mowing (is it 60 or something smaller?) and what size mower you'll use. A 4320 should be fine with a 6', maybe ok with a 7', above that you probably need more PTO power and more weight to control the heavy mower. I can tell you that mowing 5 acres with a 6' takes more time than I like, though I don't have a cab and that would make a huge difference in what it would feel like. I couldn't imagine mowing 60 acres with a 6' mower, though I'm sure someone, somewhere does that. For me it would take at least all summer and maybe two.
For other tasks, a 4320 might work. There is another person on here who uses a 4120 for real farming including complete hay work. I think he said he would buy a 4320 if he did it over again, which isn't a big jump. The 4x20 series models have a turbo and given the altitude you're at in Colorado that will be a big plus.
I have limited snow to deal with and use a rear blade and FEL bucket. If I had a lot I would likely use a front mounted blade. The snowblower will work well but I wouldn't want to crane my neck driving backwards 1700' at 3mph. If any single snowfall is more than 24" or so, that may be your only real choice.
If you want a cab - and I would (and wish I had one) - the next JD model up with a cab is a 5225, and with a FEL that will set you back at least $35k, probably more. It is a lot more tractor than a 4x20 and should last forever for your type uses. There are better deals on cab tractors between those sizes from some other makers, particularly including the Kubota M7040 and the Kioti DK55 and DK65 models. Mind you I own a JD 5105 and am very happy with it, but they aren't cheap especially for cab models. All of these would be utility tractors and HST is not available, but you can get hydro-shuttle at least on the JD and Kubotas I mentioned and that will be as good or better for most purposes (better for mowing and plowing snow, almost as good for FEL work).
I'm not sure what you mean by "not be able to handle some of the implements without help". Unless you've won numerous weightlifting competitions, you're not going to be able to pick up any implements by hand for a 4320. Nor should you need to - attaching the implements is mostly a matter of backing precisely, regardless of the tractor size. If you can clarify the concern it might be helpful.
As for haying - I don't currently do hay, but we bought with that in mind as a possible future use. Cost of operable hay equipment is significant and it requires considerable mechanical skill to keep it running. 40 acres might be enough to justify the cost, but consider the maintenance too. A 4320 should be enough to run a square baler or some of the smallest round balers, and there are several people on here doing significant hay with smaller tractors than that. For conventional round bales at a nice speed, a bigger tractor would be needed. But an operable 4x5 round baler would likely be at least $5k, maybe $10k, and would be really underused on a 40 acre field, so I'm not guessing that's your plan.