Bottom line is he put down a NON-returnable deposit. if you'd backed out of the deal 1st its your duty to give him the money back. But you didnt so you keep ALL the money.

Problem is theres no paper work to that effect. Which is good and bad for you.