This reads to me as if you are the actual Nanny here……and just wanted some advice on how to deal w/your Mom Boss walking around and that is okay.
In the event that I am totally wrong & misreading your post OP - forgive my ignorance.
Anyway it is completely normal to initially feel a little uneasy/uncomfortable having a new person > basically a stranger still at this point, right? > in your home.
Hopefully w/time as you + your Nanny get to fully know each other, your comfort level will change.
It is just that these things do take time.
In the meantime, I would stay out of their way as much as you possibly can.
Allow your baby to bond w/the Nanny which is pivotal for this to succeed.
Perhaps run some errands in the afternoon if possible.
Good luck!