If the tappets won't slide under their own weight into their respective holes don't fit them, there is absolutely no room for debris there, you won't realistically be able to tell to loose/tight as you should feel ZERO play, of course there is some but if it can be felt it's wrong.
AS the fault has occurred on just the exhaust side I would say the tappets are at fault.
If you use a 1000 grade wet and dry, with oil on it you can recover the bore most likely, but at worst a insert can be fitted, to recover it, you MUST really hard pack towelling around the base of where the tappet is first though if smoothing back.
Had this problem a few times, esp on cvh's, the last being on a "pro" built mega bucks xr2 race engine, had to actuallyforce one tappet out, then spent time lathing the tappet to get rid of bad marks, wet and dry on the bore.
tabetha