Something else to be concerned about if you go the divorce route: If you live in a community property state (I think California is one and there are others) then in the case of divorce, he is entitled to half of what is yours, and vice versa. So if you came into the marriage with $1M and he came in with $0, he will walk away with $500K.
I reiterate that it's a decision that you and you alone can make. Good luck with whatever you decide.
