Seems like two issues.

One, as a general rule should you bring leftovers into the office? I have always been in offices where it was fine to bring in leftovers - half a birthday cake, cupcakes, chips and snacks. Obviously there are some you should leave home but generally I haven't seen anyone bring something they should not have. And, if you don't want to partake then don't. Obviously some people are fine with free food.

Second issue is alerting others in the office that they were left out of an event. That is unrelated to any leftovers in my opinion. And, I think it very much depends on the event, the size of the office, who was and wasn't invited and how word gets out that some people were left out. A 10 person office and not everyone is invited to a Super Bowl party and they are all 20-something might be bad form. A large office with multiple age groups and a more formal event like a wedding, well not everyone should expect to be invited. It's still polite not to discuss in front of everyone though.

My office loves to get food. People often bring in stuff from big events they hosted over the weekend (like an anniversary party or big birthday party). I would have no idea if some people from the office were invited nor would I care. It’s not 1st grade — you don’t have to invite the whole class.
I totally don’t get why this is rude. If no one eats if by the end of the day, take it home or toss it. But most people are happy to eat the extra brownies and samosas or whatever.
I honesty feel guilt I don’t bring in party leftovers to the office. I mean to, and then somehow I end up eating the whole remaining cake myself.

I don't see the problem. There are groups of friends in my office who get together outside of the office. And there is always that crowd that loves to get the "leftovers in the kitchen" email. Personally, I don't eat food unless it's from someone I know. Also, I have my group of work friends that I hang out with outside of the office, and I really don't care about the other groups that I'm not included in.

I mean, what is it to you though?
Some people do enjoy free food, some leftover can be better the next day.

I don’t want your leftovers whether I was invited to the party or not.