We currently live in a 1200 sq foot home but the neighborhood is ideal, and the location serves us perfectly in terms of employment and schooling options.
This says it all. Finish all or some of the basement, to include another bath. To some extent it depends on how your space flows, but if this would make the difference, I would stay put. We are a family of 3 (so, admittedly smaller than yours) but our house is 1000sf on the main level, and another 600sf or so in the finished basement, where we enlarged a half bath into a full. Makes all the difference. We have two offices down there (one is office/guest room), TV room, laundry room, unfinished basement storage space, and the full bath. Very comfortable for guests and good for separating TV noise (downstairs) from living room quiet (upstairs).