Rather than expecting OP/Nanny to read his mind, her DB should have said, "Hey, I'd like some one-on-one time with Junior. Can you spend some time with just <other baby> for about 30 minutes? Feel free to take her on a walk or in a different room."
Still a little rude, but much less so than what the DB actually did. And the other child (presumably) is comfortable with the DB and maybe even happy to see him, therefore the following. DB doesn't get that, or doesn't care, fine.
Nannies care for children. We get paid extra if we're required to read adult employer's minds.