I couldnt lose the 2 nuts of the turbo at the engine side, so I took off the manifold with it. Maybe a lot more nuts and bolts, but in the end it saved me a lot of time.

Modified the manifold and turbo afterwards for quick release so in the future no difficulties anymore for me to get the turbo off.