I am 43 and have successfully spared myself that particular hell by never vacationing with friends!
We are an ADHD/HFA family. We're always going to be different, and someone might be abrupt or socially inept (and by someone, I mean my husband).
So we vacation either just our nuclear family, or with siblings or parents, who know how we operate.
I'm not saying you have diagnoses, OP. But that if you feel your family isn't as social and outwardly bRigHt ShiNy sMiLEs, that's fine. Stop vacationing that way.