I leave it in most of the time. I tow just enough it is easier to leave it in.
I have been rearended, hard, with the hitch in. It bent the hitch, as well as the cross support on the main receiver section. Truck frame was fine though. Did a job on the car that hit me... I was on my way to work, and was suppossed to pick up a buggy of concrete.
Also, I use a locking cross pin. We used to have horses. When we bought our used horse trailer, I had a simple cross pin with the hair clip holding it. The guy we got the trailer from described how he had the hairpin come loose, and then his cross pin wiggle out, while he had four horses in a larger trailer. He was on a mountian road when it happened; fortunately the safety chains caught and cradled the trailer tongue.
Ever since that sotry, I default to the locking cross pin.