When did having a mistress become a taboo?

People are almost shocked when they hear about it. It happening in so many peoples lives. Not just those of public figures.