Junkman,
When I used to do this work the main breaker panel next to the meter had it’s neutral buss bar connected to the ground rod, the power companies neutral, the branch 110-volt circuits white neutrals and bare ground wires. The breaker panels came with a green screw taped to the inside that could be run through this insulated buss to ground it to the cabinet, if as stated on the little note on the tape, it was required by local code. My boss said it wasn't so we didn't install it.
I don’t know what code says, but I would say if one were NOT to be attached to the cabinet it would be the neutral (yellow striped wire), the one that should have all the white wires attached to it.
Your description of how your installation is wired sounds backwards to me, but I haven't looked at it. I would get an experienced electrician to inspect your installation if you have concerns.