There is never any one answer to this question. It's all about your own circumstances. I just rolled over 400 hours. Probably 350 of those have been in 4WD. I have NO level ground ... none. Going downhill in 2WD was something I did once. Never again. Uphill in 2WD in not possible.
The ONLY time I'm not in 4WD is when I'm on the paved road going over to a neighbor. When I cross my paved driveway, I'm careful about how I steer, if at all. Very gradual.
A comment on brakes. These small machines don't have any. Not in the traditional sense. None of the wheels have brakes at all. The only braking is called a 'wet brake' inside the transmission and it can easily be overcome by gravity, meaning even in 4WD, stepping on the brake pedal may not help (much) if you're already rolling too fast downhill.