He has every right to if you are still on the premises, once you have left the premises and he wants to call you back then it becomes a whole different ball game. You 'should' have asked for a policeman/independant witness to observe the accusation and following evidence ( or lack of it ), then you have a case for wrongfull accusation

It all depends on whether you had left his premises or not before the accusation was made