how do you know that it's working? just that the light in the switch is on?

have you checked the electromagnetic clutch on the compressor? that must not engage if the blower is not running, otherwise the evaporator will turn into a block of ice and block your airflow into the cabin (not that you'll notice becuase your blower is switched off), and the low pressure side of the a/c system could go below atmospheric pressure and let air into the system through the compressor shaft seal. you can also damage the compressor through lack of lubrication.