It's hard to know what you're doing wrong, if anything, without knowing what you're doing. If your guests/family have no affinity for or interest in each other, then you've got a bigger problem than a lack of festive holiday feeling. You can only do so much as the host, but anything you can do to get people to talk about themselves should help to get the ball rolling. In terms of ambiance, it helps to start with a clean, tidy, well maintained environment. Neutral and sparse beats garish and cluttered, but a sense of style (any style, well done) is important to make people feel comfortable. A lot of people are pretty clueless and need help in this area.
After that, it's a matter of creating a joyful holiday mood. As a PP stated, proper lighting is important. Think table and floor lamps, and candles or votives. Relaxing holiday music. Enticing smells (that should be easy at Thanksgiving; if not, simmer water with cinnamon, cloves, and orange rind on the stove). Set a lovely holiday table or a nice buffet. Ideally, use real dishware, silverware, glasses, and napkins - not paper products. This can be done very affordably by shopping at Ikea, Target, etc. Yes, it's more work, but it makes a real difference. To the extent that you can, put out natural decorations (e.g., tree branches, leaves, berries, fruit, nice baskets, etc.), and supplement with artificial seasonal items as needed (mini lights, ornaments, wreaths, ribbons, etc.). A lot of this is common sense, but not everyone has common sense in this area.