A rotary cutter has one big blade inside a heavy-duty deck, which smashes basically anything in the way, including grass, weeds, saplings, etc. Roughest cut mower there is. The deck and blade are subject to damage from objects you might mow over like stumps, rocks, debris, etc. Can also be used as a regular mower for fields and present a decent-looking cut, though best suited for mowing of tall, high weeds and brush, etc.
A flail mower has 40-100 small, swinging L-shaped blades attached to a shaft, rotating VERY fast, creating a whipping motion of the mini flails... They won't throw large objects like a rotary cutter might, and aren't as subject to damage from large objects like stumps, rocks, and other debris. Personally, I'd say a flail mower works better if you have lighter density/hay type cutting to do on a more regular basis. Flail mower isn't going to cut down the type of brush and saplings a rotary cutter will, but it will provide what I see as a better finish cut, and do it with less noise if that's a factor.
Depending on your needs a rotary cutter might be the better all-around tool, since it too is capable of mowing a field on a regular basis and presenting a respectable-looking cut. This is why the rotary cutters are a much more popular option for compact tractor owners. That and price usually push buyers towards the rotary cutters instead of flail mowers. However, if you're after a clean cut of a field that you maintain, and don't have heavier-duty requirements, the flail mower might be worth it to you.
HTH,
