Use a 2nd (non magnetic) screwdriver to hold it in place and get the magnetic one out the way first, stops the magnet pulling it back out the grooves.