http://www.who.int/oral_health/media/en/orh_cdh_0304_05_risk_factors.pdf
This is systematic review of the literature on risk factors for cavities in primary teeth of children aged 6 and younger. Some things to note:
There is a shortage of high-quality studies of this issue.
The researchers identified the available studies that met criteria for non-bias. Few of the studies that looked at children under 3 found found that brushing less than once a day was a significant risk factor for cavities. This was more likely to be a finding in studies that looked at children age 3 and up. Also, only one study found that age of first dentist visit was an important factor in risk of cavities in primary teeth.
The final conclusion:
"Evidence points most consistently to a young child being most likely to develop caries if they acquire Streptococcus Mutans at a young age. It appears that a high level of Streptococcus Mutans may be partly compensated by other parameters such as good oral hygiene and a non-cariogenic diet.
Less than daily toothbrushing (or visible plaque) and a highly cariogenic diet are thus important risk factors, but they may interact so that if there is a balance of good and bad habits the development of caries may be controlled. Enamel hypoplasia is also a predisposing factor."
Brush your baby/toddler's teeth if you can. If you can't, try to limit sweet, sticky foods and sugary drinks. The issue becomes more concerning if your child has enamel hypoplasia or strep.