I agree that families can live in any size space (see NYC!) but in my experience, I have found that as my kids age (esp my boys) they use up so much more physical space.
They are just bigger, and gesture wildly, and talk loudly, and just seem to take up more space than when they were little. For example, my seats-six kitchen table worked when my four kids were little, but it doesn't now that they're tweens/teens. Everything is bigger - their shoes, their coats, their attitudes!
And even as teens, they tend to be where you are. They'll go off happily to their rooms for longer stretches, sure, but they always converge on the kitchen, and so the kitchen feels much smaller now that they're all bigger.
But your kids are still sooooo little, OP. Stay put for now if that works for the rest of your life. Wait to move til you just.cant.take.it.anymore. And then your new space will feel so good, and you'll be ready to make the trade-offs that such a move might entail. Good luck.