That's another thing with drying towel, any dirt that may be left, will go deep into the cloth, rather than sit near surface like a chamois.
There are a few great 'discoveries' I've had whilst getting my gear together, drying towel is right at top of the list.