We waited 4 years after our first baby to do it again, because the first year was incredibly brutal. We had some health circumstances that equated to no sleep and a year of crying, and I truly almost lost my mind on several occasions. I didn't even want to do it again after 4 years, but my husband really wanted two children, really wanted #1 to have a sibling, and we were recovered except emotionally ... I'm glad we went for it again. But we had similar health circumstances with #2 as well, and for me, it's never again.
Finances are the other big factor for me. We're very middle of the road financially and more kids would mean some big tradeoffs in provisions for the kids we already have.