Sounds like a fun project, but depending on how much dirt you have to move, I think you'll end up using the loader. Just keep pleanty of ballast on.
You should be able to get ~1/3 yard in the loader, and it is easier to load and dump when you get there. No getting off to "hoe" it out.
But if you do build it, you could use both the loader AND rear scoop like you are planning. You will almost be able to carry a full yard each trip