Hmm. Your reasons seem wishy washy. Wherever you go, there you are: you might not feel comfortable welcoming people into your new house either.
I'm the opposite. My house is 1200 sq ft, and it looks like a cute-ish cottage, but it's terribly cluttered because we're a family of 4. I invite friends over, and when the kids were little, we had dinner parties and birthday parties. Now the kids are grown (one of them is in college but lives at home in the summer), we tend to invite individual friends.
My house is decidedly imperfect and inconvenient in many ways, but I haven't lost any friends because of it
I love my house, mostly because since it's so small and the main floor is open concept, it receives a lot of natural light. I believe this has helped prevent depression during stressful times in my life.