Live-in nanny here.
You need to have a conversation with the parents and children at the same time. With a child young enough that they can't read, you need to devise a pictorial way of showing them when they can and can't come in, they need to knock, and you need a lock on your door. Keep specific hours.