I'm thinking it could be the brakes now.
Before I replaced the brakes, (disks and pads) one of the pads was cracked badly on the drivers side, which was my reason for changing them. I heard it rubbing which is how I found it was cracked after taking it all apart.
Come to think of it, I span the wheel when the old brakes were on and there seemed to be a bit of resistance, the wheel slowed down rather quickly.

If I accelerate reasonably hard, (this is when I notice it most), holding the steering wheel straight the car will start pulling to the left side, and the steering wheel isnt moving.

So I do think it is the brake on the drivers side now.
I was able to press the piston back in when I changed the pad, so I assume the piston hasnt seized? How can I tell if the sliders have seized? Is it an easy fix or would I need a new caliper? I was able to unscrew the bolts when I changed the pads and the whole caliper and frame moved fine.

Definately think its a brake problem though.