OP we have a similar situation with my DH's brother, his wife, and their son.
They live a 5 hour drive away, so much closer, but we don't have much of a relationship with them and DH and his brother aren't close: they text about twice a month, don't talk on the phone, aren't close, and we only see them once every other year or so. We don't celebrate holidays with them or do vacations together. When we do visit each other it's usually just a weekend, or part of a larger trip to the area. Our visits are only for the kids' birthday parties.
We see them every other year, which seems to work, and we keep visits short--2 to 3 days max. We have also felt like when we've visited them that they don't make much time for us and they make us feel that we're a nuisance when we visit them. So we limit visits to every other year, stay in a hotel, and entertain ourselves, really only seeing them for the birthday party and that's it.
Their last visit to us was for our daughter's birthday party. Per their choice, they stayed one night in a hotel, went to the party, and then the next day we took them out for brunch and did one sightseeing activity with them before they drove back home.
It's not the kind of relationship I was hoping for, and we have nothing in common other than DH and BIL are brothers, but we keep the visits going so the cousins can get to know each other, because their child is our child's only first cousin.
We invest our time and energy in friendships instead of family relationships, because we have no local family and the family we do have is largely disinterested in a relationship.