I wouldn't have thought the exhaust leak was the cause, but that needs sorting before anything else is done.
My car is absolutely standard. Once I was persuaded to replace the wastegate actuator. The replacement had a stronger spring and it gave symptoms similar to yours. Refitting the original removed the problem. It could still get a bit jittery on very cold damp days on the somerset levels (sea level) so I lenthened the actuator rod a turn to cure it. Previously, it had been showing 147bhp on a rolling road. Conclusion was that the boost pressure was getting higher than the ECU was happy with. Might be worth lengthening the actuator rod a couple of turns and see what that does. If it cures it, then it is possibly a boost pressure control issue.