At those ages, our only relaxing vacations were hanging out at the grandparents' house for a couple weeks (the live in a lakeside community with lots of small beaches, playgrounds, and a good pool and keep their house well stocked with toys for the grandkids) and going to family camp.
At Deer Valley Family Camp in PA the kids (age 3+) spend the morning in activities with kids their age supervised by great counselors, while the parents can hang out and do a variety of activities -- sailing, kayaking, hiking, tennis, join a volleyball team, painting or ceramics in the art center. Afternoon is for kids' naps and family activities. All meals are served family style. We first went when my kids were 3 and 4 and it felt like the first vacation I'd had in years (other than visiting my parents). So nice to just sit on the patio reading, not have to worry about what anyone was going to have for dinner.
http://www.deervalleyymca.org/