i think it all depends on what you want from a car.
I have had the m3 e46 and also the e39 m5 I have just sold. the m5 is a more refined comfortable car with a lot more "subtle" handling. the m3 is an animal that bites hard. the cosies are a hoot to drive and can be fast if tuned well, but as we will all agree, the m3 and m5 (being newer technology) will sit at 160mph all day. thrash a cossie hard enough and it will blow up.
I am looking for another cossie again, but purely because I am lucky enough to have it as a 3rd car in the garage. the m3 and m5 were wasted being sat doing nothing all week.
I am on the fence with the argument, as it does have 2 sides, but if I were cruising in my m5, no one would bat an eyelid......
cruise in a 3dr, and everyone looks!! that's the difference!!