OP I can empathize. I'm a 55-year-old very fit and energetic nanny whose current job entails caring for a 9-month-old so obviously I am extremely active all day (no TV in the home, outings every day, swimming, playgrounds, story time, play dates, sharing the whole wonderful world of a little one!).
Anyway I often encounter this because I look too old to be the mother but apparently my energetic engagement with him so often out in public has made it common for others to say things like, "Looks like Grandma is having a really fun day with grandbaby!", etc.
I just smile gently back and say, "Well he calls me Nanny but I sure do love him like a grandma!" Even if they are still a bit embarrassed by their mistaken assumption, they usually still seem to love the sentiment and that erases any awkwardness.
As for those whose sense of self-worth is so wrapped up in what other "private school mommies" might think of a nanny job title, once you have matured beyond that stage of your life (although unfortunately some never do), you will realize what a vain and silly concern that is.