My brother just shifted his 320D on so he could get a new toy. As it was when he first got it (standard) I hated the way it delivered the power and the way it drove. After a remap it was a different car altogether and was a very pleasant drive indeed. Based on what you want a 520D or 535D sounds right up your street
I'd be hard pushed to punt the M5 on though as it's one of my hankerings at the moment. Have you looked at getting something small and crap to do the mundane stuff in and then keep the M5 for best?