Everyone knows what a deposit is for, so it's tough really. Only time I'd ever give a deposit back, is if the buyer had a genuine emergency problem, and you knew they were telling the truth.

I had someone give me a £500 deposit for my engine, quite a few weeks later, he said he couldn't afford it anymore, as his missus would kick off. Fair enough, but he never asked for the deposit back