Cue all the threats of violence .

Try talking to the brother that saw it over a pint after he finishes work and just explain the situation in full how you see it. Don't leave anything out (including the joke that was taken badly).

You are then in the hands of the Gods as to what he honestly thinks of his brother, but that fact he witnessed it all, and wasn't prepared to give you any sort of warning, speaks volumes to me....