I'm not sure why PPs are dumping on you, OP. Perhaps they come from certain parts of the country that you would never dream of visiting
Do you feel that you are wasting precious free time and money on a trip to see people who, decent and well-meaning as they are, aren't that exciting?
I get that feeling with my ILs. I squelch that feeling, because they love my children and love me and can't help being from a generation and country that simply does not cater to children. They are also painfully frugal.
My advice is to accept it, lower your expectations to rock-bottom and try to eek out some enjoyment out of these trips. Your children will be fine without kid-centric activities. It's probably good for them to follow the grown-ups along for once. If they absolutely need a nap or to go to bed or eat at a certain time, then you or DH excuse yourselves and get it done.
The grandparents won't live for ever. If they're even only vaguely benevolent to your children, it's worth it.
(And don't get sucked into SIL's, or anybody else's drama.)