Pregnant with my third now, so ask me in 20 years
I think some of this depends on what was hard for you about small children. I agree with previous posters that if you’re saying “well, I just need to survive till they’re 4 and then it’ll be easier” are in for a rough time again in the teen years. I think teens are basically toddlerhood part 2, with higher stakes.
For me personally, the only part of small children that’s been really challenging is the sleep deprivation and PPD (very related, I think, for me). I just cannot function without adequate sleep and I need a lot of it. I’ve always been this way - I was still sleeping till noon on weekends in my early 30s (long after all my friends had grown out of it).
My kids are 4.5 and 3, and since I had recovered and caught up on sleep (right around my younger’s first birthday), I’ve loved parenting, and while there are certainly tough moments (toddlers gonna toddler) I’ve consistently felt like I had plenty of extra bandwidth for another. For me, tantrums, messes, endless “why?,” all over the place emotions, misbehavior, these are all perfectly manageable (if temporarily frustrating). The past two years have been, on the whole, wonderful. So for us, it’s worth it to white-knuckle through the first few months to have another. We’re also getting a night nanny a few nights a week this time!
But if you’re not feeling that extra bandwidth till your kids are 4, I would not move forward with a third.