It should pull towards the drivers side, have you got both the clips off? Sometimes you'll have to wobble the pedals slightly side to side as the plastic collars sometimes grop the centre rod.