Monday, December 22, 2014

Which Grad Program Will Be The First To Drop the GRE?

This year I am the chair of the admissions committee for the Harvard PhD in Astronomy and Astrophysics. Around the country, committees such as ours are setting to work to select the incoming class. I thought it timely to share some worries I have been having about one required element of our application: the Graduate Record Examinations (GRE) General Test, and the Subject Test in Physics.

My worry stems from three considerations:

First, I worry that the GRE doesn't actually measure the thing we really care about, namely the applicant's likelihood of success at creative and demanding research in astrophysics. So far, my search for such evidence has turned up empty: I can't find a persuasive study showing that the GRE Physics or General test scores provide a key measure of future research success. Yes, the scores do correlate with performance in graduate coursework, but to be honest I don't really care about that as an end in itself. If a student working with me finds Earth2.0, I am prepared to overlook a B- in one of our survey courses. And this isn't just my opinion: When is the last time the Hubble fellowship committee asked to see your transcript?

Second, I worry about the impact of GRE scores on the evaluation of applicants who are women and/or members of underrepresented minority groups. I encourage you to read the recent Nature article "A test that fails" by our colleagues Casey Miller and Keivan Stassun. The authors present data for the Quantitative Reasoning portion of the General Exam showing that women score on average 80 points lower in the physical sciences than men, and African Americans score on average 200 points lower than white people. I have encountered a variety of opinions among members of our community as to why they think this is the case: Some believe these scores accurately reflect the levels of preparation of the students, others thinking it is primarily the result of stereotype threat. My point (and the point made by Miller and Stassun) is that regardless of the cause, any process that effectively introduces a cut based on a minimum score can have a dramatic impact on the diversity of the applicant pool. As noted by the authors: "In the physical sciences, only 26% of women, compared to 73% of men, score above 700 on the GRE Quantitative measure. For minorities, this falls to 5.2%, compared with 82% for white and Asian people." A score of 700 is a typical minimum score used by many graduate programs in the physical sciences. (Yes, many of you may serve on committees that don't employ a strict cut-off. But how many times have you heard a committee member express concern about that low score on the physics GRE...?)

Third, the test is very expensive. Taking the General test costs $195 and the Physics exam costs $150, and each additional score report costs $27 per test. Thus an applicant who takes both tests (as we require) and needs 10 additional score reports is on the hook for $885, and many applicants spend much more if they retake a test. Yes, there is a means to request reduced fees based on financial need, but I worry that the extra burden it imposes -- and more subtle factors like the cost of traveling to a test site, or the support one's institution provides for preparation -- work to discourage some students.

For some time, my colleagues and I have been thinking about doing a retrospective study of graduate student performance to determine which factors (the GRE would just be one we would consider) best predict success in graduate school. This is difficult, because we don't so much want to compare between students in our program as compare students we admitted to ones we didn't admit. But then I would need to correct for the impact of attending a different school, and the different opportunities afforded the student, which seems impossible. (I am eager to hear, either in the comments or by direct email to me, from anyone who has tackled this.)

In summary, I am worried. I worry that many faculty members continue to believe that these tests predict success in a research-intensive program (and an applicant's ultimate ability to succeed as a scientist) but that these beliefs are unfounded. I worry that many committees effectively impose cuts that serve unintentionally to bias strongly the pool of applicants to favor white and Asian men. And I worry that we are asking applicants to spend a lot of money for a test that doesn't inform our decision making, and we are not be seeing applications from students who are deterred by the cost.

In keeping with the suggestions of the article by Miller and Stassun, our committee will be employing interviews and searching for qualities that aren't measured by the GRE, including creativity, grit, adaptability, and perseverance. But we have already gathered GRE scores and surely they will continue to factor into our decision making.

What if GREs really don't provide important information on the potential of an individual to succeed in research, and instead have caused groups of otherwise well-meaning faculty to discriminate against women and minority students in their admission process? What would it take for a department to step forward and actually not ask for GRE scores anymore?

If an evaluator of our admissions process asked why we require the GRE, how would I respond? Could I truthfully say that I had data proving that the scores were worth the cost, both in terms of diversity, and in dollars? Or would the most truthful answer be that this is more akin to a tradition? And perhaps, like so many traditions that have in the past been used to exclude women and minorities from astrophysics, this one is also worth revisiting.

13 comments :

Anonymous said...

I am in favor of getting rid of the GRE altogether. But if you are not going to do that right now, at the very least you can save applicants a lot of money by not requiring official score reports until after a student is committed to enrolling --- instead requiring applicants to simply self-report their scores. Most places already do this with official transcripts, allowing applicants to submit unofficial transcripts (or scans of official transcripts) at the time of the application and then, requiring mailed official transcripts only upon enrollment.
Nobody is going to lie about their GRE scores if it would put them at risk of having their admission revoked.

Nick Nelson said...

I am all in favor of a more scientific process in graduate admissions and I agree that there are clear flaws in relying on GRE scores, but I wonder if the other two major components of the graduate admissions process (undergraduate transcripts and letters of recommendation) are any less flawed or do any better of a job predicting success in grad school? My hunch, and it is only a hunch, is that were GPAs and letters of recommendation as easy to quantify and compare we would find just as much bias in them, along with just as poor predictive capability, as we do in GRE scores.

Nick Nelson said...

On the issue of expense, I am all in favor of reducing the cost of applying to grad school, but that discussion needs to also include issues like official transcripts (about $20 each) and application fees ($20-$100 each).

Simon Ellingsen said...

Coming from outside the U.S. system I don't have first hand experience with the GRE, but as an exercise to judge its impact couldn't you undertake the evaluation process without that information, rank your entrants and only then look at the scores? The applicants who rank highly but with lower GREs could then be give special consideration.

Nick Nelson said...

Interesting that the next post on this blog discusses the biases injected into an evaluation process due to letters of recommendation. Which metric is more flawed, GRE scores or letters of recommendation?

Anonymous said...

I am a post-graduate currently in the process of applying for graduate programs for the second time, and this article really hits home for me. Despite my extensive experience in research (including my own first author paper) I have yet to be accepted into any institutions, and this is likely due to my GRE and PGRE scores.. They're quite terrible, but I know that I can achieve success as a grad student, and the most difficult part of applying to grad programs is convincing the admissions committees that very fact. I am not defined by my GRE scores.

Jessica Kirkpatrick said...

My undergrad was at Occidental College, a liberal arts college, where they didn't offer classes in some of the topics that are covered on the Physics GRE, or where you take these classes after you need to take the GRE for graduate school applications (for instance I hadn't taken any QM before taking the Physics GRE). As a result, my physics GRE score was in the 49th percentile, which is well below the 700 cut-off mentioned in this post. I also have a learning disability, and I am sure that influenced my score on this exam.

John Johnson said...

Nick Nelson: A better question is, which metric is more disposable, the GRE or letters of rec? While letters of recommendation can sometimes be biased, not all of them are. However, the GRE is always biased, and always selects against white women and people of color. It seems obvious to me that we should ditch the GRE first, then move toward addressing unconscious bias in letters of recommendation.

Nick Nelson said...

John Johnson: I'm interested by your statement that " the GRE is always biased, and always selects against white women and people of color". How would one prove that? I think research clearly shows an average bias, but I'm not sure it would even be possible to show that the GRE is biased in every single case. I think the same is true of letters of recommendation - they show a statistically significant level of bias on average over a large population.

CyndiF said...

Try being an engineering major, Jessica! I landed in a part of the bell curve I had never seen before. Fortunately, I got into a program that took into account my GPA and general GRE scores. I don't know if I would be so lucky these days.

Anonymous said...

I am sad that this PGRE is still a thing. I just got rejected by 10 graduate programs in physics this year because my PGRE was mediocre. Compared to everything else where I had 4 years of research experience and good GPA. Finding time to study for it was almost impossible while taking the courses needed to graduate with my MS in Physics, which I did well in B-A. It was difficult finding the time to study while researching for my thesis and taking Jackson level courses.

Anonymous said...

Also, You would think that Getting a MS in Physics with graduate coursework in the B-A range while completing a research thesis would be enough to show that I can clearly succeed in grad school (on top of the 4 years of research experience elsewhere)

Anonymous said...

Einstein failed his university entrance exam, Darwin didn't make it into medicine, the psychologist Robert Sternberg received a C in introductory Psychology, Thomas Edison was fired from his first two jobs and labelled "too stupid to learn anything", the Wright brothers didn't even go to graduate school, yet they invented a functioning airplane. The list goes on... ... Who knows how many ideas or inventions have been lost to mankind because of poorly validated tests like the GRE, which may prove book learning, getting lucky on the day, or an excellent short-term memory and little else. I think by association, meaning I see solutions to problems/issues in my mind as the result of a conglomeration of data that appears in front of me. Often, I can't understand why others don't see it and I end up questioning myself. It has taken a long time to trust my instincts because of this. No written paper could measure this and certainly not a multiple choice exam.