A chamois will only scratch the paintwork if it's not entirely clean. You can remove swirl marks with a cutting compound and orbital polisher.