When you are there - - with your children, always, when Grandparents are around, you remove the children if needed.

OP, I don't get what circumstances YOU are placing your children in where you don't feel you have control.