Dont worry about it, honestly. Use some wet and dry paper around 400 grit with some WD40 and carefully sand it until its a smooth as it can be. Then bolt cam back in without followers and make sure the cam doesnt rattle about in the cam bearings. If not then alls well!

(Honest the amount of cossie heads I see with scored cam bearings, and I simply clean them up and reuse them, NEVER had any problems!)