Here is my limited experience opinion. About 18 years ago I needed to put in my own septic system if I wanted to occupy my newly built house in less than 6 months. Even though I had never even sat on a backhoe I bought a 1970s Case 580CK backhoe. My only tractor experience previous was a Ford 9N my neighbor and I bought together a couple years before. So I learned how to operate a backhoe. Then I put in the septic system and the county was very happy with it and several employees were brought out to my place so they could learn what a proper Infiltrator type septic system should look like. I also put in several ditches for water, power, and phone. I dug about 800 feet of ditches for the water, power, and phone. Then I dug out a bunch of stumps. A bunch. Then put in an extra 250 feet of driveway through the woods. Then repaired another 200 feet of road. And removed more stumps. I moved many yards of dirt and rocks. At least 8 yards of rocks and probably 30 yards of dirt. The Case 580 CK is meant to be a Tractor, Loader, Backhoe, A TLB. I could remove the backhoe but it would be a chore. Anyway, to me the Case is a beast. It is heavy and still the hoe will throw it around. I have lots of rocks and am super glad I have the thing to get the rocks out of the ditches I am digging. I also have a Yanmar YM2310 that I bought a few years ago. I keep thinking about buying a backhoe for it but I know it will only be good for digging narrow ditches of limited depth, and the rocks on my land would be a big chore for a small backhoe. It is a great machine, has a loader, and there is no way I will be getting rid of it. If I was in your position I would buy a used TLB that was plenty big enough to do the jobs you need it to do. Really plenty big enough. At the same time
I WOULD NOT remove the loader from your existing tractor, it is too useful. At least the one on my small Yanmar is. Instead I would add a bunch of weight behind the rear wheels, like a
ballast box, or something similar, to make the front end lighter when using the FEL. Not only does it make steering easier it also takes some load off of the front end which is good for the tractor. On my small Yanmar I can have more than 1500 pounds of weight hanging from the 3 point. I do and it makes a big difference. My Case is big enough that it is hard to maneuver in tight spaces, I always need to be aware of the hoe behind me. And it is pretty long. That's why having the Yanmar YM2310 is great. It is much smaller and the bucket most of the time is plenty big enough for what I need to move. So really, having both machines is great, the Case is big enough to easily handle the big jobs and the smaller Yanmar gets used most of the time for the rest of the stuff. You can always sell the bigger backhoe and just keep your present tractor, but I bet you won't.
Eric