Urmmm I don't think a brush will remove the sealant, but I can see what you mean as they're quite rough.

Try a soft brush, AG do one - or even a mitt if you don't already use one? Megs white one (MF) is quite good for tough jobs

Oh, and try drying them and spraying with a bit of QD, this will help top up protection but it won't last as long as it won't withstand brake dust heat.

Mark