I would have ignored him in the first place

How old are you, serious question? If you are 11 then fair enough you went up to him and if you are his bro at 22 shouldnt be pushing you around. That might help people answer your question