On mine, the swing arm appears to have shifted to the left (tight against right side), opposite of yours. That "spacer," essentially a solid steel bushing, fits inside the large hole on the swing arm, and there's room for the swing arm to shift right or left, perhaps 1/4". I wonder if the shifting is due to the piston essentially acting on only one side, but not sure. In any case, the swing arm shifting one way or another shouldn't be a problem. I believe your quick-hitch is assembled correctly.
Pics below of mine, viewed from the rear, as well as some pics from the Quick Hitch Assembly Manual. It shows the configuration of only the right side of the swing pivot, but the other side is identical-- from the outside: nut, lock, large flat washer, bushing, and the bolt that goes through from the inside. Hope this helps!
Also a shot of the final assembly, clearly showing the piston correctly oriented.
PS-- after using your blower for a bit, CHECK THOSE BOLTS! They WILL loosen up, perhaps the swing arm, but definitely the 4 bolts on the mounting brackets on the rear of the blower proper. I've lost bolts before!