I think he overreacted big time. I could fully understand if you guys were gone from the house for like two hours and he couldn't get a hold of you. But for ten minutes!!!! He is off his rocker. Period.
He needs to have some trust established in you by this point or what's the point of even having a nanny in the first place? What an idiot.
He really crossed the line by yelling at you OP.
I have a list of deal-breakers that I can count on one hand and one of them is if an employer yells at me, esp. in front of my charges, I am out the door. Paycheck in hand of course.
Hopefully that is on your deal-breaker list as well.
Good luck.