Removing out of balance weight can never be a bad thing.
Weight is there anyway but is balanced, some better than others, the crank is not going to go bang if you stick on a lighter front pulley, though any out of balance may be made worse in respect of vibration, though not so much as you'd even notice.
The cranks are all balanced seperately to other components as they are the heaviest rotating mass, so need to be, so you are quite safe to put on a alloy front pulley.
They didn't use damper pulleys years ago, they are only used a additional smoothness aid.
I wouldn't hesitate to use one, if I wanted to, if only for weight saving.
tabetha