If it does this regardless of which switch is used it may point to the motor playing up, when you push the buttons on either side do you hear the motor or relay clicking at all. My driver side motor packed up then a while later it worked again and it went on working and then not untill i got the motor changed.

Good luck !