Using Non-Cognitive Assessments in Graduate Admissions to Select Better Students and Increase Diversity

The following is by Dr Casey W. Miller, Rochester Institute of Technology.  The full article can be found in the January issue of Status: A Report on Women in Astronomy. 

When I became the director of the APS’s Bridge Program at the University of South Florida, I leveraged that position to raise awareness about diversity issues in physics. Thanks to many people’s appreciation of this topic, I have given physics colloquia to about a dozen departments across the country and presented invited talks at numerous conferences. Recently, I teamed up with Prof. Keivan Stassun from Vanderbilt to bring this issue even more visibility with an article in Nature. The below is intended as a brief review/resource letter, summarizing what I would present in a colloquium. 

Exercise I

Please write down the two or three attributes of your very best graduate student. It is often helpful to write that person’s name down first. Write down a few thoughts about what it was like to work with that student. Take about one minute to complete this exercise, then move on to the next exercise.

Exercise II

Assuming you’ve played along, you’re now in a position to think about a student who really didn’t work out. What are the top two or three attributes that you feel led to that situation? Take another minute to write down a few thoughts about what that experience was like for you.

Debriefing Exercises I and II

If you’re like most people, you said something along the lines of, “Student I was in the lab all the time taking data and writing papers, asked questions I hadn’t thought of — then answered them. I was always excited to get to the lab.” And for Exercise II you said something along the lines of, “Student II was smart, but gave up easily. He came running to me with questions on those days that he was actually on campus. His demeanor made being around him a drag for everyone; things got so much better when he left.” What part of your admissions process selects for I and selects against II? If yours is like most processes, the answer to this question is “it doesn’t,” or maybe you try to squirm around this by saying that you try to read between the lines in letters of recommendation and personal statements, when necessary. Good luck with that.

Some Problems

The figure below shows that racial/ethnic/gender groups have relatively large differences in average GRE Quantitative (GRE-Q) scores. These data come from ETS, the company that makes the GRE, and are in line with existing research in both education and work settings. The performance disparities of the below figure are: the same for the Physics GRE; independent of intended graduate field; the same when controlling for undergraduate GPA; the same for the SAT; the same for 8th grade math achievement tests; and the same for fourth grade math achievement tests. ETS claims these differences are related to educational opportunity/access. The significant problem facing our community is not necessarily with these scores per se, but rather with how these scores are misused during admissions. Of the roughly 180 Ph. D. programs in the AIP Graduate Programs book, ~96% require the General GRE. One quarter of these have an explicitly stated minimum GRE-Q score for admission, with the median stated cut-off being 700 (64th–70th percentile, depending on year; 155 on the new test). With respect to the figure, such “minimum acceptable” policies can have a major impact on diversity: no one under the red line would be admitted to graduate school. Any admissions system that relies solely or predominantly on the GRE will result in an admitted class that is relatively homogeneous in both gender and race/ethnicity.

GRE Quantitative score quartile ranges from 2006–2007 by race/ethnicity and gender for US citizens whose self-identified intended graduate major was “physical sciences.” The top and bottom of the marker lines are the 75th and 25th percentiles of the score distributions, respectively; the tick is the median. The solid red line represents a typical “minimum acceptable” score posted by Physics Ph. D. programs; the dashed blue line indicates the average GRE-Q score of matriculants reported by departments to the National Research Council.

Read the Full Article here in the January issue of Status.