It depends on how you like to keep your equipment. Will it break? Probalby not if you don't pound with the side sweep. It sure makes them sloppy though.
Look, the side sweep on a medium size backhoe generates a huge amount of force. You know how easy it is to slide and throw the whole backhoe around using the swing. Those swing swing cylinders are massive, the pivots are small, and the levers are long.
We all move dirt and small rocks using the backhoe's side sweep motion. We know that motion is hard on the pivot pins and bushings, but done against a moderate load it probably won't bend anything. At least not if we stop and then push....Don't ever side swing at that rock with the bucket moving. But even stopping and pushing puts a lot of wear on the backhoe boom and dipper and bucket pivots. They don't have much resistance in that direction...they were made to resist push and pull forces, not side forces.
So most of us old guys - I'm on my 5th hoe - will limit the side sweeping as much as possible. It's far better to re-position and use the hoe on a straight line push or pull.
Of course some guys just really do enjoy beating on on a machine to see what it is capable of doing. If that is your thing, then side sweep away. It's faster, and it probably won't break the hoe unless you get the side swing going fast and use it to pound on the rock.
But no other motion I can think of ages a backhoe boom, dipper, and bucket pins & bushings so quickly as using the swing to do heavy side sweeping with the bucket.
Even a newby operator can get on a hoe and tell in ten seconds if the operator has done a lot of heavy side sweeping. Everything works; but everything is loose and wobbles.