Yep...too slow. Probabally would take more $$$ to convert than just finding a used flail.
Tillers usually have about a 2:1 step DOWN gearbox. IE: 540rpm pto input makes the rotor spin ~200-300rpm.
Flails usually have about a 4:1 step-UP box letting the rotor spin ~2000rpm with the same 540rpm input.
That makes the rotor about 10x's too slow for starters.
Second, even if you did swap gearboxes, tillers are usually chain drive, which isnt as smooth as a flail, additionally flails use belts so they can slip if overloaded as a saftey feature.
Also, given that a tiller rotor is only spinning 200rpm, I doubt that they are balanced, and even if they are, probabally not to the high degree that a flail is.
Sorry for the bad news. Sure...anything is possible, and it "can" be dont. But how much do you want to spend.....