Agree with the PP. They don't know your schedule unless you tell them. And if you miss out on a few weekends together, so what? That's the way it goes. Are their kids older? As your kids get older, your weekends will fill up quickly too and you will understand better where they are coming from.
If you don't want to go on family vacations, don't go. Just let them know the schedule doesn't work for you this year and maybe next year.
And it is a jerky thing to do to your own family - DH, kids, you - to continue to accommodate their schedule all the time. Have some respect for your own family's time.