Kid friendly B&Bs in Chincoteague include Bass Channel Inn and I think Miss Molly's as well?
Hotels and condos on the Chincoteague Channel side can have nice sunset views of the channel and marshland, while ones on the east side closer to the seashore, off Maddox, can be either motel-bleh to nice with views of the water and marshes or Assateague Inlet. I;m thinking the Refuge or Sea Tag if your DH likes to fish off the balcony.

I found a nice deal on a long weekend at Bethany on Craig's List and Sea Colony units provide pool access.
In Rehoboth the only kid friendly B&B is Melissa's which is very expensive. The Corner Cupboard is closed.

Unless you are meeting up with friends at Rehoboth or Bethany, there, I would suggest Chincoteague instead. You can rent a house for a LOT less and there are lots of family friendly restaurants, a grocery store, ponies to look at. We set up camp at the beach with beach blankets, 2 umbrellas, a cooler, a nursing coverup & extra towels and baby actually napped at the beach so we were able to stay most of the day...you will see a lot of baby carriers on the beach there! The crowd is a little less fancy, but there are a lot of DC tags in the national seashore parking lot.
Recommend Sea Star for great sandwiches and fruit salad, Gary Howard if you want to take home steamed crabs or cook your own local seafood, a a gratuituous trip to Mueller's for ice cream at their old school Victorian shop. Memorial Park has a nice playground with bucket swings.

You could do something like the Atlantis Inn-- they even have some units with a kitchen? We once stayed at the Bellmoor with a small child-- it was nice but a pricey (and still kind of a schlep to the beach).

Sea Colony has condos. You can easily do weekends in the off-season, but during summer they may require a full weekly rental.

We have been to the area many times before the baby, but would usually stay at b&b's, which aren't exactly child friendly. I don't think we can afford to rent a house in Bethany for the summer. Can you rent houses in Bethany for just a weekend, and any idea how much they run? Any other suggestions for a small family? Thanks!