I have zero experience using a flail mower as a hay cutter. The key will be can you adjust it or alter it somehow so that is discharges the hay in as big of pieces as possible without mincing it up. Flail likely not best choice for leafy types of hay.
I do have experience using a bushog (rotary cutter) as an improvised hay cutter. It actually worked fine as far as the quality of the grassy type hay goes but you will suffer yield loss using even a rotary cutter for hay. I can only guess that my yield loss was somewhere in the 33% range with my bushog (an ole squareback) but some have experienced even higher loss rates. The key with a rotary cutter is to also discharge the clippings without mincing them up (tilt the back up or sometimes even remove one side of the bushog). Bushog likely not best choice for leafey types of hay either.
Any improvised hay cutter will suffer yield loss in 2 ways. One, is you are running over the hay first with the tractor before you cut it so you will not cut it all since some gets mashed down (what does not get cut can cause issues when raking too). Second, is any improvised cutter will likely produce some smaller clippings that are too small for the rake and baler to pick up. Only you can determine how much yield loss is acceptable to you. On a hobby hay operation the yield loss is usually peanuts and will not justify buying a better tool on just a few acres, but as the acres goes up then you might want to rethink....
If you need every morsel of hay or you are doing this for $ then there really is no substitute for a proper dedicated hay cutter. Lots of choices out there and each has strengths and weakness depending on type of hay producing and also dependendant upon tractor you will power it with.