I farmed 188 acres of my own and as much as 75 acres of rental ground for over 35 years. I bought and sold used tractors, using some of them along the way. But for the most part, I took care of the place with tractors smaller than 70 hp. (67/40/47 at first, then 60/40/47hp most of the years) For a very short while, I had a 110hp mfwd tractor that was simply overkill for the amount of ground I farmed.
Nowdays, I run a mowing business. 40 to 45 hp/6' mower combinations will cover approx. 2-1/2 acres per hour on average. 60hp/8' will cover 3 to 3-1/4 acres per hour. (In average conditions, relatively mild terrain) I now own a 95 hp tractor w/ 15' batwing mower also. 6 to 8 acres per hour in most conditions, SOMETIMES more. Mowing at speeds over 5mph is very much out of the ordinary. Besides being unsafe, results suffer. The exception would be mowing with a disc mower. They cut nice at higher ground speeds.
No one says you have to mow the entire place, start to finish, all at one time. I kept my pastures mowed with a 6' or 7' mower. These days I live on 45 acres. I still mow with a 6' mower and 47 hp tractor. I never spend more than 5 or 6 hours at a stretch, mowing the place in stages. Even at that, the place is always well cared for.
If you have aspirations of someday getting into doing your own haying, settle for NO LESS than 45 hp for small squares, and 60 hp (min.) for future round baling. You can "get by" with less, but that's exactly what you'll be doing.... getting by.
I grew up on a farm too. My father took care of his 265 acres primarily with a 32 hp tractor. (I still have it)
With 100 acres, you have room for bigger equipment/higher hp. But taking care of 100 acres with medium sized equipment isn't at all out of the realm of possibility.
Now, for personal opinion. If I was starting from scratch today, and I had 100 acres to tend to, I'd look at something in the 60 to 70 hp range, MFWD most likely but not an absolute. Implements for that size tractor aren't too outragously expensive. You should be able to find good used implements at a reasonable price. I'd be looking at utility tractors as opposed to compacts. Personal recommendation based on what's out there in todays market? Look at the Deere 5X03 series. A lot of tractor for the price. (No, I'm NOT "Deere only".)