OP your son doesn't want the classroom bullies at his party.
He has to interact with them at school and after school clubs. Why should he be forced to deal with the bullies at home and on his birthday?
In my experience with kids this age they often will downplay how bad the behavior really is. Listen to your son.