Hi OP - I am going to echo the experience advice. Past behavior is predictive of future behavior. Look for a nanny who has a work history of several years at a job, one who has previously worked with children of similar age to yours, ask behavioral interview questions (some have been previously suggested in this thread) and check references in detail.
Good luck. If the budget permits, you may have best luck using a reputable nanny agency as they have tons of experience sorting through the applicants.
The International Nanny Association also publishes a free eBook The Nanny Employer's Handbook that may be helpful?
http://nannyassociation.org/ina-nanny-employer-handbook/