It sounds as if you don’t have kids. Believe it or not, you caught a break. Demand he go to couples therapy with you. That therapist can work with you both , or might have him see an individual therapist as well. If he refuses therapy or only goes through the motions, leave him. If he cannot shut up about a fantasy that hurts you, he has a serious problem. And, eventually, he’ll have that three way-with you or without you.