I'm in Giles county just south of you. I too bought the farm two years ago but do have a lot of past experience in various farm activities. My heifers just dropped their first calfs, sure are pretty.
That being said, they do take a fair bit of time. However, I know other farmers that get away with little effort and seem to do okay.
I had a JD 5300 before I moved here. First, a FEL will be the most used part of a tractor. Not a day goes by I don't use it for something. I'm an equipment junkie and would like to own all the toys mentioned above, but the wallet won't support that habit. SSL are nice, but a good heavy tractor and loader will do you fine. You said you didn't have any tractor experience so I heartily recommend getting a new or almost new one through a local dealer you can depend on. Farming is tough on equipment and unless you are a pretty good mechanic, service and repairs are something you will need to get help with.
My farm has pretty good slopes. The 5300 is 2WD and they are not forgiving to novice operators. For both safety and performance, I recommend a utility size with 4wd. The added weight and stability will give you much better performance for dirt work, and just about every other chore. The compacts are okay, wish I could add one for mowing and raking hay, but if you are going to do with just one machine, I think you will find the compacts lacking. (no offense to those that have one, like I said, sure wish I could add one!).
The horsepower you select will depend on how much time you have to get everything done. I'm getting by with 55 hp for haying, Bushhogging, grading, plowing, and just about everything else. However, I can barely keep up with all the chores and make the improvements the farm needs. Yes, you can Bushhog a 100 acres with 40 hp and a 5' cutter, but it will take you over twice as long as a 10' cutter on 80 horses. If you are maintaining a full time job, you better revise your farming plans using low horsepower.
I totally agree hay making is expensive and poor return for small acreage. BUT - the first time you can't get your hay in when it needs to be done - you will probably change your mind. Start out small with going halfs on the hay, add some equipment a little at a time and gain experience too, without a huge initial outlay.
Make friends and be a good neighbor. Most of the farmers around you will help and give you good advice. They know what works in your area, where to get deals on equipment, and will be invaluable in case you need another pair of hands. Renting out the pastures the first year might be good way for you to develop a good relationship and many free lessons!
Sorry for the long post. Since I bleed green, my recommendation is a JD 5000 series, 75 to 100 hp. If you plan on doing hay, I'd err towards the 100 side, but no less than 65 if you are going to round bale. I pull a 4x5 and I have to be really cautious of down slopes.
For the bigger jobs, either hire heavy equipment or at least rent. You can get a 650 dozer for a week from Columbia for $1600. That would cleanup a lot in short order.
Good luck and please be careful.