My parents live a 9.5 hour drive away and flying there is difficult (no direct flights, really expensive). They drive to our house to visit usually 4x a year and stay for about a week each time. We drive to their house 1 or 2x a year and stay for about a week each time. Then we usually go on vacation to another destination with them 1 or 2x a year also.
My in-laws live a 3.5 hour drive away. They drive to our house often, about once every month-6 weeks or so, but only stay for a day or 2 at a time. We drive to their house 2 or 3x year usually for a 3 day weekend.
So we see my in-laws much more often than we see my parents but over the course of a year, we actually spend more time with my parents than with my in-laws.
How often to visit depends on: your relationship with the grandparents and how much you all enjoy the visits; how you feel about the traveling portion (I like road trips so I don't mind the long drive to see family); how much vacation time you get and whether or not you enjoy spending it to see family or would rather go elsewhere (I love to travel but right now since our kids are small I much prefer using vacation time to visit family); how comfortable it is for you to stay at their house and how hospitable they are when you visit; why they are unwilling to visit you (if it's because they have health issues or need to be close to home for some legitimate reason or have an allergy to your dog or something like that, I'd be much more understanding than if it's just because they don't like traveling).