The party is for YOU and your guests. The kid is going to be clueless. Make sure your child gets all the sleep he/she normally needs--don't cut into naptime, if that means your child is going to be massively irritable. Don't take offense if friends can't come because your party time interferes with their child's naptime.
I suggest: A short fete around your normally scheduled meal time. For us, that was dinner with family members + cake. The other kid got a pajama-pancakes party starting at 8am for whoever was in the neighborhood and wanted to hope over with their kids in jams and robe.