Loaders and backhoes have a way of being useful even after you finish your primary job. Obviously you don't want to buy them if it breaks the bank but on the other hand I suspect they keep a large percentage of their purchase price when you go to sell or trade your tractor.
Remember that to rent and transport equipment is a hassle too. Rather than try to cram all your BH jobs into one day or weekend, if you have a hoe you can do the jobs at your leisure. If it costs $5000 for a backhoe that you use for a couple of years and then find you don't need anymore, there will be plenty of buyers willing to spend $3000-3500 for a BH like that with homeowner hours on it. Your real cost is therefore more like $1500-2000 and that is what should be compared to the $250/day plus hassle factor for a small excavator.