A University Wants to Assess Bias in the Classroom. Are Student Evaluations the Best Way to Do It?

Two professors surprised their university over the weekend with an op-ed in The Wall Street Journal about two perennially sore subjects: faculty evaluations and perceived political bias.

A committee of administrators, faculty members, and students had endorsed adding three questions to Villanova University’s course surveys for the fall of 2018, under the heading “diversity and inclusion.” The questions asked students to evaluate their instructors’ sensitivity to bias in the classroom. But the language in the questions seemed like a test of political opinion, wrote Colleen A. Sheehan, a political-science professor, and James M. Wilson, an associate professor of religion and literature.

“In short, students are being asked to rate professors according to their perceived agreement with progressive political opinion on bias and identity,” they wrote on Friday. “At a minimum, all charges of insensitivity, injustice, and bigotry will become part of the faculty’s permanent record.”

Villanova pushed back on Monday, denying that the questions were meant for evaluating faculty members. “Rather, the questions are designed to enable our faculty to understand how their students perceive their interactions,” wrote the Rev. Peter M. Donohue, Villanova’s president, in an email to the campus, “so they can create an unbiased learning environment for students from diverse backgrounds, social identities, and political beliefs.” Portraying the questions as a “political litmus test” is “untrue,” he and the provost added in a message sent later to faculty members.

The dispute highlights the clash of perceptions that can arise when forces like politics, free speech, and racism collide in the classroom — and how evaluations can themselves reflect those forces.

Villanova, which dealt with its own reports of racist harassment after the 2016 presidential election, has sharpened its focus on its own institutional biases in recent years. Students, in particular, pushed the administration to deal with bias on campus and in teaching. The only way to fix the problem, the thinking went, was to measure and understand it. But the method — course evaluations, which Villanova calls the Course and Teacher Survey — has some faculty members, liberal and conservative, worried that students’ responses could be used against them.

The controversy took hold online, first in conservative spheres of academe, then with a broader range of academics worrying that the survey questions could invite biases of all sorts.

Two of the questions asked students to rank on a five-point scale whether an instructor “demonstrates cultural awareness” and “creates an environment free of bias based on individual differences or social identities.” A free-response question asked students to “comment on the instructor’s sensitivity to the diversity of the students in the class (for example, biological sex, disability, gender identity, national origin, political viewpoint, race/ethnicity, religious beliefs, sexual orientation, socio-economic status, etc.).”

Measuring those issues is common and necessary, said Ken Ryalls, president of the IDEA Center, a nonprofit focused on university assessment and evaluations. “On the face of it, it’s certainly not inflammatory.” Including diverse perspectives in teaching, after all, is “kind of the foundation of liberal education,” he said.

But the specificity of Villanova’s questions may be “dangerous” when tied to feedback on individual faculty members, Ryalls said. “If your faculty are very uncomfortable about this, ask it generally,” he said. “Don’t put it on a course evaluation. Ask it of the student body in a separate email, or a separate survey.”

Or, if kept on a course evaluation, Ryalls said, make the questions more general. In a course survey his organization offers colleges, for example, students rate how an instructor “helped students to interpret subject matter from diverse perspectives (e.g., different cultures, religions, genders, political views).”

‘Oppression Within the Classroom’

Villanova conceived of the questions about two years ago. Student activists were involved early on, meeting with administrators to discuss the questions and larger “systemic” issues, according to Patrick Flynn, a senior. “Part of what made our push successful was grounding the move in student experiences of racism/oppression within the classroom,” Flynn said via email.

The university piloted the questions in the fall of 2017 across about 7 percent of its course sections. About 2,300 sampled students answered questions about 126 instructors, according to Craig M. Wheeland, vice provost for academics.

Formally, the Faculty Congress and its Academic Policy Committee, which includes students, faculty members, and deans, advised the administration about the questions’ wording and the pilot survey, said Jerusha O. Conner, an associate professor of education and the chair of the faculty congress when the diversity questions were introduced.

The rollout wasn’t flawless, Conner said. Some students, surprised to see the new questions on their surveys, thought their instructors may have been targeted as subjects of concern, she said. “We shared those concerns with the administrators.”

The questions went campuswide in the fall of 2018. Faculty members would be able to see their results once that semester’s surveys were done, Wheeland told faculty members by email, in September. “However, the results from the Fall 2018 administration for these questions will not be used for purposes of any type of faculty evaluation (annual or triannual),” he wrote. “Instead, we will use the Fall 2018 administration as another opportunity to fully ‘pre-test’ the questions in order to ensure they are effective.”

Sheehan and Wilson weren’t reassured. They saw an opaque, “progressive move to institutionalize these questions,” Sheehan said. “This is not just stifling one point of view. This is stifling inquiry.” Wilson emailed Wheeland with his concerns, Wilson said.

“Based on the faculty members serving on APC and the positive feedback from the pilot test, I think a large majority of faculty see these questions as helpful to them,” Wheeland responded, referring to the Academic Policy Committee. “But we’ll learn more through this Fall’s administration and can determine if adjustments are needed.”

The administration said on Monday that it is continuing to gather feedback from instructors and students about the questions, and analyzing the responses. “Indeed, we have been quite pleased to note that students at Villanova overwhelmingly rated their faculty at the very top of the scale,” wrote the president and provost.

Villanova allows tenure-and-promotion committees to consider only five questions from the Course and Teacher Survey. The diversity-and-inclusion questions are not among them. Instead they score rigor, learning outcomes, and the quality of instruction.

But even those metrics aren’t without controversy. “There has long, at Villanova, been concern about how these CATS evaluations are used for evaluation, especially those five questions,” Conner said.

Instructors didn’t just worry about political bias. Outside research and Villanova’s own data showed that women and faculty of color were consistently rated lower than their white male peers, Conner said.

Villanova’s data also showed that students of color had a lower sense of belonging on campus than their white peers, she said.

Conner was “very proud and pleased” by the president and provost’s response on Monday to Sheehan and Wilson. But “if I had the power, I would not use any of the CATS questions for evaluative purposes,” she said.

Instead, she’d support something like Bryn Mawr College’s student-faculty partnerships, in which student consultants gather holistic feedback about instruction.

Conner is optimistic about Villanova’s next steps. The university is measuring its diversity, equity, and inclusion efforts in all sorts of ways, forming new committees and running campus surveys, she said.

“That’s a key focus as we do our new strategic plan,” Conner said. “We’re centering diversity and equity like we never have before.”

Follow Steven Johnson on Twitter at @stetyjohn, or email him at steve.johnson@chronicle.com.