Its part of the bush, and does need to be removed prior to fitting new bushes.

And yes it is not easy to get out