As a nation, we have no shortage of opinions about race-based affirmative action. This spring The Wall Street Journal published an op-ed by a high-school student wondering if she had been rejected by the Ivy League because “I offer about as much diversity as a saltine cracker.” More than 1,200 readers commented. By the end of the week, she had been invited to appear on the Today show.

As I write, we await the U.S. Supreme Court’s decision in Fisher v. University of Texas at Austin, which could limit affirmative action. In March the court announced that it would also hear arguments in a second affirmative-action case, Schuette v. Coalition to Defend Affirmative Action, which will decide if voters in Michigan were within their constitutional rights when they approved a ballot measure banning the use of race in college admissions. By taking on those two cases, the court seems to be signaling that it believes something is not quite right about how we use race-based affirmative action.

Whatever the court decides in those cases, it’s clear that at some point, our national debate over race and affirmative action shifted from a primary concern over when and how colleges might use the policies to help students overcome past and present economic, social, and cultural barriers to a belief that such policies should be used only if they don’t keep middle-class white students from attending the college of their choice. In the process, we have ignored the fact that the fewer numbers of black, Latino, and Native American students there are on a college campus, the greater the likelihood that white students will racially harass them.

Those were the findings of a June 2012 research brief issued by the University of California at Los Angeles’s Higher Education Research Institute, which found that on campuses with the lowest diversity, racial harassment is a consistent and growing feature of college life. That should trouble us.

According to The Journal of Blacks in Higher Education, since February there have been numerous instances of racial harassment on campuses, such as Elizabethtown College in Pennsylvania, where 19 incidents were reported in a one-month period. The college is 88 percent white. In March the liberal and historically progressive Oberlin College suspended classes after a series of racial incidents during the previous month brought the campus to a standstill. Oberlin’s student body is 72 percent white. A month before, students at Emory University were forced to apologize after they aired a student-produced television show asking viewers to identify “which students on campus shouldn’t be here and are only at the school because of affirmative action.” The segment went on to make light of lynchings, tarring-and-featherings, and cross burnings. According to admission information, while 41 percent of Emory students are white, only 15 percent are black or Latino.

In one of the most disturbing signs that low diversity on campuses is a problem, a group of students at Towson University calling themselves members of the White Students Union announced in March that they would begin nightly patrols to protect white students from “black-on-white crime”—even though college officials say Towson has one of the lowest crime rates in Maryland. When asked why he had formed the group, the organizer responded that “diversity is not strength.” Whites make up 68 percent of the students at Towson.

Hopefully we have not actually reached the point as a country when we would agree that overwhelmingly white campuses are more desirable than racially diverse campuses, or that white-student aggression toward black students is a matter to be taken in stride.

Moreover, although we rarely mention it when discussing affirmative action, racial harassment has an impact on more than just students. In May the first and only black professor of head-and-neck surgery at the University of California at Los Angeles sued the university for racial discrimination. His accusation? That he was depicted as a gorilla being sodomized by his white superior in a slide presentation to the medical-center staff.

I don’t think we fully understand what it is about the presence of certain racial groups that leads to instances of harassment by whites. But we do know that some groups covered by affirmative-action policies have not been subjected to the same kind of continuing challenges to their admission to college. Just consider the fact that since 1967, when affirmative action was extended to women by executive order, that group has probably benefited more than any other. Preferences for women have not triggered anywhere near the kind of legal controversy as racial preferences have. And while there have been affirmative-action cases about employment issues, centering on male plaintiffs’ claiming to have been discriminated against in favor of female workers, we have yet to have such a case reach the Supreme Court that focuses on college admissions. In the discussion about affirmative action and higher education, race matters most.

Indeed, according to a 2009 poll by the Pew Research Center, support for the idea of affirmative action differs widely depending on how the question is posed. When respondents were asked if they broadly supported affirmative-action programs to help blacks, women, and other minorities get better jobs and education, 70 percent said they did. However, support dropped to 46 percent when people were asked if such programs should give blacks preferential treatment. Put clearly, the vast majority of people agree with the idea of affirmative action in the abstract, but have an issue with black students’ specifically benefiting from it.

Looking at that finding in light of the increase of racial harassment on predominantly white campuses, we can only conclude: As long as there are too few minority students to stand out, some white students feel as if there are too many to tolerate.

Which brings me to a recent incident that I have come to believe is a metaphor for the ignorance, fear, and hysteria dominating cultural conversations about race and affirmative action. Some weeks ago, my husband and I were on the way home in our Prius when we stopped at a local store to buy a few things. I stayed in the car.

A few moments later, the back door opened, and someone threw some unbagged groceries in the back seat. The next thing I knew, a young white woman jumped into the driver’s seat and tried to start the car, which just requires pushing a button on the dashboard. I was initially too stunned to say anything, but when she noticed me sitting in the passenger seat, she panicked and began swatting at me, asking what I wanted, and shouting at me to get out of her car.

I was just at the point of getting it together to tell her that she was sitting in my car when she jumped out and started screaming for help. Of course, she began to attract lots of attention, and a few people were making their way over when, literally in mid-scream, she noticed that her car (also a Prius) was parked behind ours. She stopped screaming, opened the back door, took out her groceries, jumped in her car, and drove off without a word of apology.

I can’t know for sure, but given the frequency of racial profiling of drivers by the police, or the much-publicized fear of black people invading middle-class homes, I was left to wonder if her reaction would have been the same if I had been a white woman.

Though confusing one car for another is something that might happen fairly regularly, the extreme escalation and overreaction of the young woman has come to represent for me a microcosm of how an African-American presence can expose fears and reveal underdeveloped racial attitudes both in academe and beyond.

It is time for us to be less compelled by the stories of white people who feel they have been robbed of what they are entitled to, and more concerned with the students on campuses who are suffering racial harassment. If we take those students seriously, perhaps we can begin to focus on troubling the conscience of the many, instead of easing the discomfort of a few.

Noliwe M. Rooks is an associate professor of African studies and feminist, gender, and sexuality studies at Cornell University. Among her books is White Money/Black Power: The Surprising History of African-American Studies and the Crisis of Race in Higher Education (Beacon Press, 2006).