Sounds exactly like a problem that's happening to someone else I know, they had the same issue with a fence that fell down, guy A) Approaches neighbour and asks about mutual costs to fix fence, he says "F*ck off", so guy A) gets fence repaired, then neighbour complains to police etc and sloictors are involved saying it's substandard blah blah blah, so he ripped it down and left it a mess, then built one further back in his own garden and left the guy with the mess. Sweet! TBH that kinda sounds like what's happened with you seeing as you have a fence in your own garden.
Doesn't sound like you'll get anywhere without hassle, personally i'd just go round and tell him to mind his own fcuking business and that'll be then end of it as far as your concerned.