If you actually did agree to a social meeting w/each other, plus your current partners, it would be sort of like lying if you didn't disclose to your new partners what you two shared in your pasts.
Sure, you could do it & see if you could get away w/it, but it seems deceptive and no one wants a relationship based on that.
So you would have to let your current S/Os know the deal. And once they did, it would be hugely awkward. Tenfold.
Best to can the idea altogether. I am sure you have more than enough "friends" you can socialize with than this guy.