Idle means just that. I would have a go at cleaning the throttle body, as from you're description it might be a sticking butterfly valve inside it.