I was the youngest. When I was younger, it was a little harder in the sense that I was less coordinated, so I wasn't as good at sports, and less emotionally mature. But, the upside was that I pushed myself to develop the skills that the older kids had. By high school and college it really only mattered in the sense that I got my driver's license a little later, or turned 21 a little later. Otherwise, there was nothing negative at all.
I would have been really bored if I had been held back, since academically I was more than ready. My grades were always great, I went to good colleges and graduate schools, etc. As an adult, it matters not one bit.