if it's not switched on there'll be no drag on the engine. the compressor has an electromagnetic clutch to disengage it, the only slight drag is from turning a pulley with a bearing in it.

potentially more to be gained by losing the weight of the system, but is it really worth it? i'd rather keep it for demisting.