There are 4 type of Women in a man's life that will benefit him and allow him to experience the 4 points of the Magic Compass.
North, South, East & West, in life.
There are also 4 negative Women who lead a man astray into the lesser directions of NW, NE, SW & SE.
You Dave, played your Card North with your first marriage and then forfeited your ticket South with your second marriage to experience North (Relationship) again. No wonder you are feeling something lacking in your life (that journey South). Sadly, you just have to suck it up for what it is.
James Bond never slept with Miss Moneypenny. She was his ticket South (Money) and kept him employed.
Dante Alligjeri's Beatrice was his South (muse) that lead and inspired him to write the Divine Comedy. He married another woman as his North and was thus balanced.
10,000 of the richest and successful men were asked the secret of their success? Their top answer was "Never go into that Motel room" (with that woman of the South). True survey.
You my friend, via your second marriage kinda did so.
You turned what would have been something 'different' in your life with the woman of the South, into just another North (relationship) experience.
Alas, for you and many men (who sleep with the Secretary, who would have otherwise lead them to the wealth of the South direction) - it's too late. You can't undo it, even by trying to sabotage your second marriage. Because, everything happens in threes and you'll only forfeit another Card by turning your East or West woman into a 3rd North experience.
So don't freak about your connection with your first Marriage, while married again in the North zone. It is what it is - the emptiness you feel for never having experienced what going South could have been for you. You can't get it back. It's about understanding what JaSin (of Ur) has just shown you and being at peace with your choice of North and then North again, in your direction in life for lack of North & South (and what it could have offered you and your original marriage). Be a man, understand this and once you own this. Trust me, things will be better to a lesser degree sure - but better for it none the less.
I appreciate a nice $10,000 counselling fee for this advice.
Please contact my Secretary of whom I don't sleep with, even if my marriage is no more.

Now a word from our sponsor.
"Word".