I've been excavating a medium sized pond with my FEL since last summer, so I've gotten a lot of practice with numerous different conditions. I have a B7800 (30 engine hp).
If a pile of dirt and/or rocks is loose enough and not too steep, you can start from the bottom. As said by others, a toothbar and some heavy
ballast on the 3ph help with digging effectiveness and stability. With tractor in 4WD and bucket level and at grade level I drive slowly into the pile until the bucket is as full as possible; then curl max back and lift slightly. Keep the load as low as your terrain will permit, particularly if you are going to be moving over uneven ground (you want to keep the center of gravity as low as possible for max stability).
If you can't curl the bucket or lift it, it is probably because the matterial is too packed or the pile so steep that there is too much weight above. This is almost certain to be the case if you have been excavating undisturbed earth and are attempting to cut into the vertical side of the excavation. In this case, you will probably have to raise the bucket so that it cuts into the side not more than 1 1/2 bucket depths below the top of the grade; you should be able to curl and lift the bucket fairly easily but lower it as soon as you have backed out enough to be able to do so.
I have been able to remove even fairly large (up to about 8-9 cubic feet) boulders with my FEL. It helps to nudge the boulder into either a slight rise or other obstruction that will offer some resistance while you drive the bucket blade under it as far as you can. Even with a fairly deep boulder, if you can get the blade and bottom of the bucket under well more than 1/2 of the boulder's bottom surface, you should be able to get it all the way in the bucket when you curl it back.
With big boulders, you may want to give some advance thought to the direction you load it from, so that you will be able to get the boulder as deep into the bucket as possible. If it is not in the best position for loading, roll it by nuding it with the FEL into a better position.
Last but not least, I have been able to move even boulders that are so large that my FEL cannot curl or lift them (over about 1000#) by running the FEL in as close to them as possible and then running chains from the hooks on the top of my bucket around the bottom front of the boulder and back to the other hook and then dragging the boulder where I want to put it by backing up. It is also possible to push/roll a boulder forward if you can either curl its bottom up with the FEL as you are pushing it or place the blade in an irregularity near the top of the boulder and then push it. I have used these methods for removing some humongous boulders from my pond excavation that my FEL couldn't curl or lift at all, including a couple that were close to, or perhaps a bit more than, a cubic yard.
I hope that some of this helps; a lot of the fun of using a FEL is experimenting to learn what works to solve particular problems.
