Is there a way to safely determine what angle the tractor can go up before the accident happens?
Extend the backhoe outwards so it will catch you if you tip. Then drive up steeper and steeper hills until it happens. Then use the backhoe to push you back down.
PS: DO NOT DO THIS. I'm just kidding.
All kidding aside, it seems like there are too may variables to be able to say 100% "yes this is a safe angle, but this isn't." Let's say you parked the tractor on a turntable and slowly tilted it until it tipped over, and you determined that 18 degrees was safe. Well, what if you're in the real world and you hit a gopher hole that dips one wheel and takes you over the edge? What if you've got a bucket of dirt? What if you've got a bucket of compost? What if you've got a bucket of WET dirt?
I'm not an experienced operator, so I just take. it. slow. whenever I'm in a remotely sketchy situation. Rollovers happen very quickly when they happen, but if you are moving slowly, chances are that you will get some warning before everything turns pear shaped. Your front end may start to slide downhill just a bit, requiring you to turn uphill to counteract (WARNING! DANGER!). You may go up on three wheels for a second (WARNING! DANGER!). Your front end may get a little bit bouncier than it usually is going up a hill (WARNING! DANGER!). And so forth. The slower you go, the less likely you are to end up upside down without any warning.
The other mistake that I think inexperienced operators make is ignoring or missing the warnings. If you start to get warning signs, pay attention. Don't just put the pedal down and trust in your guardian angel to make everything all right.
Bottom line: stay within your comfort zone. If something doesn't feel right, stop. Assess the situation. Determine the best course of action, and proceed slowly to the nearest safe position. Determine how you're going to get from point A to point B before you start going, don't just charge in and get surprised. Once you know an area real well, you can tear around it willy nilly if you want to. If you were to see how I get from some points on my property to other points, you might think I was taking the long way around, but what you don't know is that the obvious way has a steep hill, or a bunch of ground-hog holes, or a slippery muddy spot, or whatever. My way may be longer, but it keeps the tractor with all four wheels down, and that's what matters most. No job is worth not going home at the end of the day.
If you get to the point where you are a very experienced operator, you may come to ignore some of these rules, but from your question, you sound inexperienced like me.