Hey OP, I live in Victoria

Don't even ask how I found DCUM!
May and June it starts to warm up, but there will be rain a few days a week usually. July to very early September are generally quite warm (20-25 degrees most days) with little rain (I think it rained 3 or 4 days max this past summer, and maybe a few more cloudy days). If you come June or later don't worry about the weather/rain. Victoria especially is lovely - I read about summers in DC and am amazed at how different it sounds. Ours will be warm but not overbearing, little humidity, way fewer bugs!
All of Vancouver Island is gorgeous in the summer - I highly recommend Tofino/Uclulet (even if you're not into surfing, just for the beauty), Saltspring Island (great daytrip with kids but you need a car for sure), Royal BC Museum in Victoria. To the west of Victoria (about an hour's drive) there are amazing beaches on the other coast. Tons between Sooke and Port Renfrew. Highly recommend French Beach (great kid-friendly hiking) and especially Botanical Beach - if you take any of my recommendations Botanical Beach is definitely #1! You can easily spend 4 hours hiking around and exploring.
Maybe I'm biased (

) but I really disagree with 23:59. Victoria is a tourist-oriented city and to say there's not much to do suggests that they didn't do much research. May-September most of the city is completely ruled by tourism, cruise ships, tours, etc. It is also a central location - Vancouver, the rest of the island, and Seattle are all within a few hours.
I can't speak with as much detail about Whistler other than it lives up to the expectations and is a phenomenal location, especially in the summer for a family that likes the outdoors and hiking.
I don't know if this has been helpful or not but feel free to ask any other questions you have! I'm familiar with almost all of Vancouver Island/BC if there were any other places you were wondering about.