Been an age old problem this.
Always tell them it's "really awkward" to find your house,satnav won't pick it up etc,do a bit of research first then offer to meet said person away from your house and drive them to the location if it's a non runner/no Mot etc.
It can be tough though cos the old boys will think something's up if you get a bit evasive about where you live etc.