Using easy start wont make an engine harder to start when you are not using it, in fact sometimes it can make it easier to start as its quite a good cleaner of the inlet tract so in the case of carbs with sticking pistons due to gunk etc (ie a mini SU) it can actually help fix a problem.
If it doesnt start without it, its not addicted to anything, its simply not fuelling or sparking correctly.
Your view is correct, you are ignoring a problem which is getting worse, so thats why it gets harder to start without easystart.