A marriage, or the equivalent for same-sex couples, is in part a vow that you will maintain a monogamous lifestyle with your partner as long as you are both alive. (Or, more accurately, it should be.) That is a vow that you and your partner make to each other before witnesses, including your friends and family, an individual authorized by the state to conduct marriages (justice of the peace, priest, rabbi, etc), and (if a religious ceremony) before whatever God or gods you believe in.
You could argue that the adulterer broke his/her vow to his/her spouse, thereby hurting his/her spouse, and that therefore the adulterer should be punished. I'd think that any punishment meted out by the courts (at least for a first offense) would be something along the lines of community service, rather than being locked up long-term (or stoned to death
).
If there were a punishment for adultery, I think some people would do it anyway, but it might deter some others.