Wapster had a 500 until recently. Never heard him say it was horrible to drive once.
I can imagine that due to the lag it can be hard to drive fast round town. But i dont think they are horrible to drive if you dont have to have it on boost all the time. Motorway and long streches of road would be awesome to wind a 500 up on!