Yes mate, it all has to come out. we chopped it out with a screwdriver and hammer. Once you start breaking it up, it comes out quite easy though.

Tip for you though, when pressing the new bushes in, be careful because the edge of the bush carrier chops into the bush material.
We found that if you wrapped a large jubilee clip around the leading edge of the bush and compressed the lip enough to get it past the edge of the bush carrier, it slides in beautifully. use plenty of the supplied grease.
The side of the bush with the cut-out faces the wishbone body. It not obvious, but it would be a real pain to get out if you fit it in the wrong way