torque is not affected if you maintain the same engine rpm as the same air/fuel is being burnt pushing the same piston down the same stroke acting on the same crank. the difference is that, particularly at low engine speed, any load from trying to pull away will reduce the engine speed more as you don't have the same inertia behind it. it just means that you have to use the throttle more to maintain engine speed which can be difficult to control precisely as the engine is quicker to respond.
ime the positives far outweigh the negatives.