OP, I hear ya. I am in the opposite situation. My DS just started kinder this week via early entrance and is the youngest in his class. He is turning 5 next week while others are turning 6! It is not an easy decision at all. If we had let him go on time I think we would be in your shoes. Instead, we chose to try it out and I am having the opposite worries about if we made the right decision. Are we setting him up with a disadvantage? Is his emotional immaturity going to affect his entire academic career? etc. I think we can't win for losing you know? All we can do is try out best and we will always wonder if we made the right decision. At least we are loving parents who are trying to do what is right for our kids.
I also know of another mom in your shoes. Her son passed early entrance too with a Sept bday. She chose to not let him start kinder and had him do an additional year of pre K. Well, when he got to kinder his reading was so advanced they moved him to 1st anways. I spoke with her and she still worries about the decision to move him to 1st grade now since he is the youngest and more immature, etc. I swear there is not a perfect answer to this dilemma except to not have kids in Sept!!
Maybe when they graduate we should all have a glass of wine and share war stories

Good luck to you. Maybe you can challenged him with additional reading etc at home, or after school programs in addition to speaking with his teacher. Best of luck!!