The problem is that the tests narrow the curriculum. The students from higher SES families will have lives that are broadened due to outside activities, trips, camps, music lessons, discussions and reading at home, etc. so they will not suffer as much as the lower SES students who get most of their "education" at school. But, overall, all students suffer from the focus on test scores.
Problems go far beyond that. The poor performing kids are spread out whenever possible because the average scores are what matter.