if you dont want it coming back then you will have to make sure its all cut/grinded out.

dont think your being fussy at all tbh. pointless job if you dont do it properly, unless it just a tidy up for a track car/to sell etc