Lose torque??!!

That's not true, what you will lose/gain (however you want to look at it) is momentum.

Just as a light flywheel will allow the engine to rev faster, it will also make it lose revs faster when you come off the throttle, meaning that you may fall out of the powerband on gearchanges.

Mine has run a marginally lightened flywheel for years and to no ill effect. I have never been able to measure any gain, but its like anything, its hard to prove/disprove.