It kind of sounds like he isn't super happy about it. I don't know what sort of arrangement you two have but if you do decide to go ahead,it would be a nice gesture to make a deal with him that he also gets to have another woman if he wants to.
Perhaps he has fantasies or things he wants to do...