Assuming you compare two rather similar cars then this is what I think:
A 2wd is more fun to drive, in the dry it's a little bit faster, but it's also more demanding.
A 4wd is less fun, but once the rain drops start falling it's so much faster and easier.

As to being faster: a 2wd maybe faster in a straight line. On a racing track I am not sure it's faster - even in perfect dry conditions.
I have driven fast 4wd Sierra's on track days and the thing I remember is: Ideal line for cornering? O yeah, that's where everybody else is when I am overtaking them.