Having them the wrong way shouldn't make any difference, your stereo just won't hold its memory.

Have you got amps wired into the car by any chance