The 4x4 exhaust manifold locates the turbo slightly higher than the 2wd manifold. Usually this means that the 2wd downpipe is closer to the front anti roll bar, and may indeed catch, causing annoying vibrations when driving.

Lack of sealing of the downpipe on the turbo suggests that the sliding joints of the exhaust need looking at, or the mounts are letting the exhaust hang badly. Take the exhaust system apart and start fitting it again, starting with bolting the downpipe onto the turbo.