Yes it's sexist, though I probably wouldn't say anything to FIL. I would definitely hand the baby to BIL in your situation. I would probably find a parent or MIL if it were only me and FIL in the room. If I were alone in the house with FIL and the baby, though, I would probably change and feed the baby.
Relatedly, I stayed with my parents for a couple of weeks when DD was 2 m.o. I wanted to take a shower, and I left dD with my Dad (mom was not at home). He asked me nervously what he should do if she needed a diaper change. I reminded him that he raised 3 children and, if my mom is to believed, was fairly involved in basic things like feeding and diapering. And I also assured him that diaper technology hasn't changed substantially since the 70s and 80s, so I felt confident that a highly skilled medical specialist like himself could probably figure it out. He did end up changing a diaper while I was in the shower, and we all lived to tell the story
