Harm the car?

Driving her Passat, it's quite easy to change with out even feeling it, especially 3rd to 4th and back.

I do it all the time, any chance of long term damage? Obviously yes if it's crunching and all sorts, but how about when you hardly feel the change?