I'd recommend your own home, with DH cooking. Mother's Day is one of THE WORST times for eating out in terms of crowds, plus everyone brings their kids, so it isn't exactly a peaceful experience.
If you insist, Del Ray Cafe has good brunch and is family friendly while staying fairly classy. It has a French flair, although it isn't typically a buffet. I think there are a few places in Old Town, too, but again I think they're mostly sit down, not buffet.
We're going camping.
