Can't say that I do. I think a persons attitude/personality has a lot to do with how I'm attracted to them. Even if there was a strong physical or sexual attraction, don't think I could see past the traits that are annoying me. That being said, I don't find most people annoying.
Assuming it was a one-time deal, would you and your partner consider an offer like the one in Indecent Proposal?