Do what ever makes you happy and live with it.

You should never be with one woman because it is the right thing to do or for the fact that you have a child.
Everyone deserves the right to be happy and if you do not love your missus and want to be with another woman you should leave as your missus deserves the opportunity to be with someone who truly adores her.
Likewise you should never ever stay together for the sake of the kids, they can tell when you are miserable and it effects them, you can be a dad without being in a relationship with their mother.

The biggest question to ask yourself remains is this lust you are feeling or something more ?