Sounds like the majority of the time his wife is with another woman, not another man. And as irrational as it is I believe it would be a lot more acceptable, and even exciting, for most men. Especially since he's invited to the party.
In my opinion there are a few reasons why open relationships are just plain harder than monogamous ones.
1) Both parties have to be completely on the same page, and let's be honest how often does that happen between a man and a woman both sexually and emotionally?
2) Being swingers I get, having sex with someone else is understandable. Having two relationships with two different people at the same time? That's nearly impossible for anyone. It takes way too much time and effort and opens the door for all kinds of issues. Regardless of how "ascended" above human emotions you think you are, you're fooling yourself if you think a situation like that is ever completely stable. Especially the way The Master goes about it. Everyone is sleeping with everyone and 4 people are involved in orgies with common frequency. No person on earth can completely separate themselves emotionally from the physical. Because if the sex is devoid of emotion it's lifeless, cold and boring. Lust is the strongest driving factor in desire, and lust gives way to jealousy easier than any other emotion. All it takes is for you to break up with your girlfriend and your wife wanting to remain friends with her after a bad situation.
Anyway, I understand the idea of open relationships and don't begrudge anyone involved in them. But I also understand the idea of monogamous ones. Do whatever works for you. I'd like to try an open relationship just because I want my wife/girlfriend to bring home hot women I can also have sex with, for no effort expended on my part. However, I know I'd fail miserably at it. And I sure as shit would never be able to have a wife AND a girlfriend. One relationship is plenty.