My parents came up from Florida and my sister lives nearby. We invited the godparents as well as a few families with whom we're friendly with at church. It was a simple lunch after mass. And of course there was cake.
You don't have to have a party. You can go out for a really nice lunch with your immediate family and celebrate with your son. Even if just the grandparents came, that would be perfectly fine.