Cape May, for sure.
Rehoboth has more oceanfront rentals (in the manner of OBX: new-home giant construction with 9 decks, 7 bathrooms and ivory vinyl siding.) So there's a point for Rehoboth. The beaches are all free, another win for RB.
But all other indicators point to Cape May: the beaches are more plentiful and more beautiful. They are 700% less crowded. The town is prettier and much more extensive. There is mini golf, and it's adorable.
Speaking very very specifically about a 4 year old now, the experience at Wildwood (an 11 minute drive along a stunning coastline) is incomparable. Funland is super sweet, and we went all the time when DS was 2-6 y.o. But, Morey's Piers in Wildwood -- esp. the Surfside pier -- have more and better rides for a **4 yr old** than even Disney. I am not making this up. You know, the 'baby rides'.
The boardwalk in Wildwood is louder and much more obnoxious than that in Rehoboth. I wouldn't hang there with a young child though -- I'd just leave the amusement park and drive back to my cute place in beautiful Cape May.