Depends on what you want to do. As you imply in the question, not only hp, but also weight of the tractor is important. Balers & hayracks & moco's are heavy, & you need a heavy tractor to stop those items.
The older square balers (JD & NH are sometimes more dependable) require 15 hp or so to spin the baler (the flywheel does the hard work...) and another 15 hp to pull the tractor & baler through the field. A total of 40 hp on a good heavy tractor can bale, & pull a moderate sized hayrack with 130 bales or so. You really want some type of live pto. Yes, it is possible to bale with lighter tractors etc., but you want to know what works, not what is on the fringe, right? Then the modern high-capacity square balers need about 50 hp just for the baler, you'd like a 80-100 hp tractor on the baler.
A small round baler can get by with 45 hp, again you need some weight as you are spinning 800 lbs or so of bale! Many round balers prefer 80 hp, and the big boys can make a 125 hp tractor really work.
To cut the hay, about 20 hp will run a sickle mower, about 40-50 hp will run a haybine (aka moco) and you need some weight on the tractor to control it, and there are bigger outfits that need more hp. There are various spinning cutters that need 25-80 hp.
Tedding or raking hay is pretty simple, 20 hp is plenty.
It would only be the biggest CUT that you could consider putting a baler or haybine on - and that would be borderline. Most of the midsized ones can handle a sickle mower & raking.
--->Paul