Like the weights mounted on the front weight bracket in the picture?Rob- You may be losing rig balance removing your FEL. I would not see this as a serious problem on level terrain, on hills and slope you may want some weight low and up front to stay upright.
I usually drop the loader for mowing. It’s just so much better. Visibility is way better, it rides a lot better, you can get a lot closer to objects and it unloads the front enough that the steering brakes actually work.