Who's side were the trees on?

Who maintained and pruned the trees?

Do you own the house or is it rented?

As stated check the deeds if you have them, you could either put up a fence or plant more trees, up to you. If the trees were on his property its his shout, I agree its out of order for him not say something though.
Trees always cause domestic disputes, its quite a popular one especially when they grow tall and become an issue with maintenance and blocking out the sun