When he was invited by his cheating friend's wife, he said he really wanted her