The fling with Joan was just that, a fling. They spent a weekend together. It happened only once. He didn’t develop feelings for her and never wanted her to leave her husband. Peter was heart-broken(you did the breaking up) and looking for company. Joan was simply a distraction. He told you before the two of you got back together. He didn’t want secrets between the two of you.
The reader didn’t date anyone seriously. You went out for a drink with a co-worker. Peter saw the two of you together. That’s the night he met Joan. He thought you had moved on and it broke his heart.