It has started getting very cold so we can only do so much indoors... I would want to take the kids to the library, museum etc but I can't.
Different parents have different boundaries and desires for what their kids do during the day. If this family wants their kids to stay in the house or play in the background and you knew this going in then I doubt that you will change their minds. From the parent's perspective, young kids don't really need these outings. It makes the job 100% more interesting for the nanny but its really more for her than them. Some parents are OK with this and others are not. You just have to find a job that fits your expectations.
We had more of an infant nanny when the kids were infants. She didn't drive so it wasn't an issue. When we replaced our nanny (for reasons not having to do with not driving), we did seek someone who could drive. Since the kids were preschoolers/older toddlers, we wanted them to go to the park where they could play with lots of kids or the library. I don't care if the nanny chats with other nannies as long as the kids didn't get hurt or lost. If the kids were infants, I would care that the nanny was chatting with other nannies while the kids were sitting in strollers at the park. There just isn't much for an infant to do at a park so IMO its not unreasonable to restrict activities at the younger age.
I did draw the line at going to the mall, fast food places or driving on the beltway/more than 15 minutes from the house. We take the kids to museums and the zoo on weekends. My nanny does not like driving anywhere near DC so again no issue. However, if she decided to wanted to take them to a museum for a change of pace even though we take them on weekends, I would not be OK with it. Its actually less about trust and more about why expose to driving in traffic and on the beltway when they don't need to be there for someone else's benefit not their benefit.