Not sure what level of fancy you're looking for or what type of food, but we have had wonderful brunches at Blue Duck Tavern with our kids. They have coloring pages and crayons, and the staff is incredibly nice to (well-behaved) kids. Be sure to make a reservation.