chamois wont scratch anymore than anything else if they are PERFECTLY clean, but, they do tend to marr the paintwork.

drying towels are alot softer and wont marr the paint !

If you dont want to correct the paintwork with a compound polish the only way to lessen the swirls would be with a polish ( non abrasive ) that has alot of fillers. a test was done at what was best at "filling" the scratches was auto glym super resin polish.... but it wont last long, dependant on how often and how you wash your car.