There is an optimum point for spooling the turbo for each cam

More advanced than optimum = more lag
More retarded than optimum = less lag

I dont know what optimum is on each cam on your engine, or what your current timing is, so I cant comment if yours will get better or worse if you advance it with any 100% certainty, but I would *expect* it will get better if you advance it a little if currently its as retarded as it sounds as typically the standard marks arent too far out for relatively early spool.

Also you are considering the two cams together, you will find that the optimum almost certainly has different values on each cam, so really you need to swing each one around one at a time to find the optimum for each.

Bear in mind that you will probably then end up with the extra torque you were trying to avoid though, so will need less boost lower down to avoid it, so will need a boost control setup that will do that.

You will also probably lose top end power, as generally retarding the cams will move the power up the rev range like you have previously been correctly informed.