I have the same brake set up as you and 1 of my rondells just slightly touched the caliper too, the other 3 were close but didn't touch

try another wheel & see if its the same

If you need some spacers let me know, I can sort you some 1mm spacers