Rider,
I'm sorry I didn't see your post until just now. I just replaced my belt and though I went through **** I did not have to remove the flywheel. Please read my entire post, even if you do not have the same problem I had I think you will find this information useful and will probably not have to buy an impact wrench. My gut feeling is that you may not have read the fine details found in the owner's manual concerning the belt removal/replacement procedure. I've included a copy of that page from my manual at the end of this post.
It could be that your idler arm is "stuck" like mine was. That actually can be good if the square hole they cut for the breaker bar overlaps the pulley like mine does (no way can you get a breaker bar in mine without damaging the pulley). Because of that I had to remove the spring from the idler arm first (not fun). Then I had to pound the arm with a hammer to get it to move to a position where the belt was loose.
I later determined that the idler arm pivot bolt was too tight from the factory and a slight loosening allowed it to move like it should. Also, my idler pivot bolt is a regular bolt and does not have the grease fitting like the one shown in the pictures you displayed. I had to move the idler arm until the pulley was just touching the left transmission breather tube before the belt was loose enough to remove (it was still a b@tch 'cause the belt was still so tight).
The MOWER manual provides the finer details about this procedure not found in the repair manual. You have to first pull the belt UP over the engine pulley (counter intuitive, huh) that loosens the belt enough for you to pull it UP from the idler pulley and then up over each of the transmission pulleys. Then the belt is loose enough that you can pull it down from around the engine pulley. Some kind of wrong engineering went into this design, at least for maintenance purposes!
If the square hole on you idler arm is cut properly then you should be able to use a breaker bar to leverage the idler arm so that the pulley touches (or almost touches) that transmission breather tube. If you find you can't move the idler arm then you might check your pivot bolt and if it does not have a grease fitting then try barely loosening it from the lock nut, no more than a 16th or 32nd of a turn.
I'm not sure what I'm going to do next time, since my idler arm is now functioning properly and I can't get a breaker bar in there 'cause they cut the hole inside the pulley's diameter.
Mine is a 2006 model but I imagine yours is not that different. Here is the page from my manual that covers the removal/replacement of the drive belt: