That's why I grew a backbone. There are situations when the more, the merrier, but sometimes I just want to spend time with close friends. If everyone brought their own friends I didn't know, the dynamic would be completely different.
Yes, it's rude to ask. People who don't agree are also the ones offended when the answer is No.