I think it depends on the child and the museum type. My 1yr old and I were regularly in museums pre COVID, several times a month. Some of those were toddler specific monthly and weekly events. A science museum with large dinos and other animals to view, maybe yes, unless baby is scared. An art museum with flat pictures on the wall, maybe no.
The toddler classes taught parents how to interact with babies in the museum, flat photos..try identifying certain colors..or recurring animals.