Why Caltech is now requiring test scores

Forum Index » College and University Discussion

1 2 3 4 5 6 7 8 9 ► Go

Go to page...

08/12/2024 14:13

Subject: Why Caltech is now requiring test scores

Anonymous

Reason behind Caltech reversing test blind policy. Makes sense and the data is not surprising.

https://tech.caltech.edu/2024/04/26/letter-sat-reinstatement/

Faculty petition
January 16, 2024

Dear faculty colleagues:

Over the past few years, faculty colleagues across campus have noticed and commented on a sharp decline in the quantitative skills of our undergraduate students. In particular, although many of our undergraduates are of the same caliber as in the past, there has been a concerning drop in preparedness at the low end of the distribution. This decline has worsened with recent changes in our admissions practices, and is particularly acute for the current sophomore class. An inordinate number of students are failing courses, honor code violations are on the rise, and requests for tutors and extensions have substantially increased. Some faculty report having to adjust grading practices, as well as course content, to the change in student population.

We fear that this decline will have disastrous consequences for our students’ training and career outcomes, for Caltech’s educational mission, and for Caltech’s reputation at large.

The goal of this letter is to initiate discussion and action on this critical and urgent matter.

Below we consider possible causes for the decline. Based on these reasons, we believe that the problem requires both immediate action as well as longer term improvement and monitoring in admissions practices.

In the immediate term, we ask the institute to:

Reinstate the SAT/ACT as an admissions requirement for the next cycle. This should be announced in March 2024, so students can start testing in the spring, preparing for applications in the Fall.
Form a faculty-led committee to study the effectiveness of current admissions practices on student outcomes and to make recommendations about how to improve the process. Such a longitudinal study was promised to the faculty (see Faculty Board meeting of 6/7/2021), but no such report has been released. This committee should report its findings sometime in Fall 2024 so that it can help shape next year’s admission cycle.
In the longer term, we ask the institute to:

Establish a faculty-led standing committee whose charge is to regularly gather data on student learning outcomes and use them to evaluate and guide our admissions processes. This is important because so far our admission policies have not been informed by this essential exercise, in contrast to peer institutions like MIT.
Why do student STEM skills matter?
Many of us are committed to Caltech because of its unique place in the higher education landscape, as reflected by the “There is only one Caltech” campaign motto. We view our educational mission as recruiting, educating, motivating, and empowering the next generation of top scientific, engineering and mathematical talent. Our comparative social contribution is to provide a niche for individuals with an extreme passion and talent for these fields. We give those students a protected environment to develop their talent and passion at the highest levels of science and engineering. Then they go and establish the semiconductor industry or find a cure for AIDS. Historically, Caltech has produced one of the highest rates of future STEM PhDs and the highest rate of Nobel laureates. If we give up on the goal of educating students with this unusual intensity and talent, then we lose our raison d’etre, our unique and essential educational contribution to society.

As faculty, we also need to acknowledge the limits to what we can do. The historical greatness of our undergraduates has been largely due to them, not to the faculty. To train top-flight scientists and engineers we have to start with top-flight high school graduates. Our skill is in designing a curriculum of courses and research that challenges these students beyond their comfort zone. But we have no special skills that would bring the median high-school graduate to that level. There is nothing magical about Caltech that turns someone into a successful scientist just because they spent 4 years here. Furthermore, unprepared students struggle here even though they would have thrived at other top schools like Stanford or Harvard. This is why the STEM skills of our entering first-years continues to be crucial to Caltech’s unique educational mission.

Two examples: Student performance in EE44 and EE55
The drop in STEM skills has been observed by many faculty who teach first-year and sophomore courses.

A concrete example is provided by Ali Hajimiri, who analyzed grades in EE44 (Deterministic Analysis of Systems and Circuits). EE44 is the introductory circuits course taken by all EE sophomores, and it uses basic complex number, linear algebra, and calculus concepts. Ali has taught EE44 continuously for the past 12 years. Each year, he administers a basic math test on day 1 to get a baseline on the students’ math competency. He also administers a midterm and final exam. This fall, he reused the 2020 final exam to create a control comparison.

Consider the scatter plots below, which show the relationship between the score in the initial math Quiz 0 and the midterm exam (red dots) and final exam (blue dots). Each dot represents one student. There is a stark difference between the grades of 2020 sophomores (left plot) and 2023 sophomores (right plot). Whereas the top of the class in 2023 (green ellipse) looks similar to the entirety of the class in 2020, the class of 2023 has a sizable cluster of students (the red ellipse) that did not exist in 2020 and who enter the class with weak math foundations and in turn performed poorly in the course.

Another data set is from the EE55 class (Mathematics of EE) taught in alternative years by Victoria Kostina. This data compares the final exam scores of the students taking the exam in 2021 versus. those taking it in the fall of 2023. It again shows a noticeable drop in the performance of the class.

Although this is data from only two courses, it is consistent with the classroom experiences of many other faculty at Caltech. If, as we suspect, the data from other classes at Caltech match these observations, then we are facing a major challenge to our educational missions that requires urgent action. First, a substantial fraction of the current Caltech student population is not well matched to our educational program and not served well as a result. Second, the experience of all students is impacted, for example, by lowering the level of our course offerings. Third, our reputation, and thus our long-term ability to attract Caltech-caliber, students are at risk. Eventually, this could affect recruitment of graduate students and faculty as well.

Decline of Caltech’s performance in prestigious student competitions
Historically, our students have had an outsized presence at the Putnam math competition, with multiple Putnam Fellows (top 6 finishers), and topping the competition more than any school other than Harvard and MIT. But since 2010 there has been a steady decline in Caltech’s showing. Over the past few years, Caltech’s performance fell precipitously: since 2019 we have had zero students in the top 100. This is distressing for a school that touts itself on being a destination for top STEM talent. MIT, on the other hand, is sweeping the top spots.

A similar decline relative to other universities has been seen in coding competitions, such as the ACM-ICPC, where in the past few years Caltech has even failed to qualify for the international competition (before that it was a contender for the top spots).

While we are not suggesting all Caltech students should be top math or coding competitors, our performance in these competitions provides an informative signal about the quality of our student population, and gives us visibility to help attract top high school talent.

Potential causes for the decline in student STEM skills
Several hypothetical causes for the drop have been proposed. We hope that the faculty-led committee that we propose will carry out an immediate quantitative and systematic evaluation of these issues to inform our admission practices.

Here we provide an initial discussion of two of these causes.

Is it fully attributable to COVID? This explanation fails on two counts. First, the top half of our student population performs as well as the pre-pandemic students. Given our large pool of applicants (~16,000), and low admission rate (~2% for non-athletes), it defies reason to think that we cannot find more A & B students and have been forced to admit D & F students to fill the class. More likely, our admissions process is failing to spot the D & F students. Second, the COVID hypothesis does not explain the differences in top achievers across schools. COVID or not, top Putnam performers still exist. They are just not at Caltech.

Is it caused by changes in admissions practices? Our admission criteria have changed in the past few years and thus deserve scrutiny. Starting with the class entering in 2021 (today’s juniors), as a response to Covid, we stopped requiring applicants to take the SAT/ACT test, which in the past was used as an indicator of math and verbal proficiency. Furthermore, we introduced a number of non-cognitive criteria alongside academic merit. In the process, we seem to have lost focus on the need to choose applicants who have acquired in high-school the skills needed to thrive in Caltech’s rigorous and fast-paced academic training.

Why bring back the SAT/ACT as soon as possible?
The case for using the SAT/ACT in our admission process is that it provides a necessary, but not sufficient, signal for success in our challenging educational program. These test scores are unlikely to be predictive of outcome differences at Caltech among students who perform above a high-threshold, as has been the case for our historical student population. However, based on years of experience in the classroom and the lab, we believe students who are not able to score highly on the math sections of those tests are not likely to perform well at Caltech.

Consistent with this view, in March 2022, MIT brought back the SAT/ACT as a requirement [ref1,ref2,ref3]. The report from the MIT dean of admissions is well sourced, and — given the similarity of MIT’s mission to our own — makes for useful reading. Here are some relevant quotes:

“Our research has shown that, in most cases, we cannot reliably predict students will do well at MIT unless we consider standardized test results alongside grades, coursework, and other factors. These findings are statistically robust and stable over time, and hold when you control for socioeconomic factors and look across demographic groups. And the math component of the testing turns out to be most important.”
“It turns out the shortest path for many students to demonstrate sufficient preparation — particularly for students with less access to educational capital — is through the SAT/ACT, because most students can study for these exams using free tools at Khan Academy, but they (usually) can’t force their high school to offer advanced calculus courses, for example. So, the SAT/ACT can actually open the door to MIT for these students, too.”
“[T]here is no pathway through MIT that does not include a rigorous foundation in mathematics, mediated by many quantitative exams along the way. So, in a way, it is not surprising that the SAT/ACT math exams are predictive of success at MIT; it would be more surprising if they weren’t.”
Similar results have been found by several recent studies at other institutions [ref4,ref5,ref6, ref7]. For example, a study by Opportunity Insights looked at admissions records and student outcomes at multiple college Ivy-Plus colleges between 2017 and 2022 and found that “[e]ven among otherwise similar students with the same high school grades, we find that SAT and ACT scores have substantial predictive power for academic success in college,” even after controlling for high school grades. As shown in the figure below, “[s]tudents opting to not submit an SAT/ACT score achieve relatively lower college GPAs.” A related earlier study by Opportunity Insights also found that SAT/ACT scores are substantially more predictive than high-school grades of the likelihood of attending an elite graduate school or working at a prestigious firm.

In stark contrast, three months after the MIT announcement, Caltech announced that we would extend the moratorium by three years. In fact, the press release from admissions making this announcement stated: “…standardized test scores have little to no power in predicting students’ performance in the first-term mathematics and physics classes that first-year students must take as part of Caltech’s core curriculum. Further, the predictive power of standardized test scores appears to dissipate as students progress through the first-year core curriculum.” This claim refers to an internal report that has never been released to the faculty for evaluation and discussion.

In fact, the predictive value of the SAT on Caltech student performance had been studied in the 1990s by Dave Rutledge and colleagues. They found that students with a Math score below 700 have a high chance (larger than 50%) of dropping out. In the wake of that study, the admissions office set 700 as the minimum Math score for admissions.

As recently as 2019, all of our admitted students had an SAT Math score above 700, with the 25/75 percentiles at 790/800. In fact, historically, Caltech students had the highest SAT scores of any university. Now our admission process dismisses the SAT as a useless metric. One of the tenets of empiricism is that extraordinary claims require extraordinary evidence to support them. Given that this claim goes against a practice that has served Caltech and MIT well for decades, that MIT recently looked carefully into this issue and brought back the SAT/ACT, that recent studies have found that SAT/ACT are predictive of student outcomes at Ivy-Plus colleges, that the 1990s Rutledge study found similar conclusions at Caltech, and that the report cited by the Admissions Committee Chair has not been shared with the faculty for evaluation, we are skeptical of the claim that it is not a useful metric on admissions.

Vol. CXXVII, Issu