Once here, once there. Or two max visits if someone wants to double up. Mutual vacations elsewhere are good too.

OP - DH can go visit his own parents. Alone. They want to see him. He's THEIR baby. Yes, they want to see the grandkids BUT if he's not willing to travel with the kids without you, that's not happening. Otherwise, let him know, more than once a year, for the time being is it, and you don't want to hear about it either.

We have similar distances to get to the grandparents. We have 20 days of leave. Each side gets 5 of those days, whether we go to them or we take the time off while they come to us. One side involves 4 expensive plane tickets and a rental car. We’ve done that trip only 3 times in 9 years and never with kids under 3. Too much gear for a flight and way too much hassle with the time change.

That leaves us 5 days for our own family vacation and 10 days between the two of us to cover school closures, sick kids, and our own doctor appointments.