You have to pull it straight back, it takes a bit of force. Just remember the fog light / heated screen switches and wiring will still be attached so don't get too carried away.
If you have a telescopic magnet then it will make retrieving the little metal clips so much easier.