It will alter the engine all in positive ways APART from smoothness lower down, this is where heavy comes into it's own to dampen the vibrations caused by trying to rotate a device(crank) iregularly, it firing stroked every so many degrees apart, of course this can't be helped, but that's the primary reason they are so heavy.
It will accelerate quicker much more so than the weight you remove would have you believe, it will also allow for quicker gear changes(if the box is quick enough) by not maintaining the higher speed of the previous gear so easily due to less inertia.
Downsides are few but are all vibration related, and only at idle, very low rpm.
tabetha