Last summer in these pages, Mordechai Levy-Eichel and Daniel Scheinerman uncovered a major flaw in Richard Jean So’s Redlining Culture: A Data History of Racial Inequality and Postwar Fiction, one that rendered the book’s conclusion null and void. Unfortunately, what they found was not an isolated incident. In complex areas like the study of racial inequality, a fundamentalism has taken hold that discourages sound methodology and the use of reliable evidence about the roots of social problems.
We are not talking about mere differences in interpretation of results, which are common. We are talking about mistakes so clear that they should cause research to be seriously questioned or even disregarded. A great deal of research — we will focus on examinations of Asian American class mobility — rigs its statistical methods in order to arrive at ideologically preferred conclusions.
Most sophisticated quantitative work in sociology involves multivariate research, often in a search for causes of social problems. This work might ask how a particular independent variable (e.g., education level) “causes” an outcome or dependent variable (e.g., income). Or it could study the reverse: How does parental income influence children’s education?
Human behavior is too complicated to be explained by only one variable, so social scientists typically try to “control” for various causes simultaneously. If you are trying to test for a particular cause, you want to isolate that cause and hold all other possible causes constant. One can control for a given variable using what is called multiple regression, a statistical tool that parcels out the separate net effects of several variables simultaneously.
If you want to determine whether income causes better education outcomes, you’d want to compare everyone from a two-parent family, since family status might be another causal factor, for instance. You’d also want to see the effect of family status by comparing everyone with similar incomes. And so on for other variables.
The problem is that there are potentially so many variables that a researcher inevitably leaves some out. Sociologists typically adjust for covariates like age, gender, race/ethnicity, region, and education, but other, unmeasured covariates (like the varying tastes or preferences of individuals or groups) can confound the relationship. This is known as the problem of undercontrolling, which can lead to omitted variable bias. The danger: A researcher can reach a conclusion that shows an effect but offers no certainty that the independent variable studied is responsible.
For example, alcohol drinkers received bad news recently when a new study came out correcting earlier studies that showed the health benefits of moderate alcohol consumption. It turns out that those earlier studies were probably measuring the effects of other habits of moderate drinkers, such as healthy eating or exercise. Alcohol has nothing to do with it, according to the new research.
Let’s say you want to find causes of inequality. Education level is an obvious one, but you want to find others, such as marital status or racial discrimination or something else. So you need to “control” for education in your regression analysis — you want to compare people with the same education.
But even when controlling for things like education, you typically don’t have any information about other important variables like personality. You are not parsing out all the personality traits that college graduates are more likely to have (greater motivation, greater ambition, more reliable and punctual behavior, better social skills, less engagement in deviant behavior, etc.) because data on the latter are usually not available in the same data set. These aspects of personality are correlated with obtaining a college degree, and employers might actually value them more than any particular knowledge learned during college. In other words, it might not be the college degree that matters so much, but the personality traits held by those who chose to seek a degree in the first place.
The opposite sort of problem occurs when researchers overcontrol. Sometimes they add too many control variables into the multivariate equation. This error can occur when the researcher is motivated to show that some variable of interest isn’t “really” important. For example, suppose that a sociologist hopes to show, statistically, that growing up in a single-parent family has no effect on a child’s educational achievement in terms of grades. The sociologist could control for family income, time spent studying, school absences, and behavioral issues in school. However, these factors might be at least partly affected by growing up in a single-parent family in the first place. By overcontrolling, the sociologist will be successful in generating only a small statistical effect of a single-parent family structure. Combined with misleadingly cherry-picked data, over- and undercontrolling is a recipe for shoddy, unreliable social science.
Consider The Myth of the Model Minority (Routledge, 2008), in which Rosalind S. Chou and Joe R. Feagin assert that “when researchers have examined Japanese and other Asian American workers in comparison with white workers with similar jobs, educational credentials, and years of job experience, the Asian American workers are found to be paid less, on average, and are less likely to be promoted to managerial positions.” But Chou and Feagin’s claim lacks any serious empirical evidence. Instead of directly citing published research from peer-reviewed journals, Chou and Feagin instead based their conclusion on a book by Tim Wise, a self-proclaimed “prominent antiracist” journalist who has no graduate training or any professional experience with evaluating labor market or socioeconomic data.
Wise, in turn, references an old newspaper article published in The Washington Post in 1992 — and summarizes it incorrectly. The Post piece exaggerates a few informal comments made by a demographer referring to some exploratory cross-tabulations of education level by race and income. An outdated and statistically naïve newspaper article thus becomes the sole evidence that Chou and Feagin reference for their conclusion that Asian American workers are underpaid. Meanwhile, Chou and Feagin ignore many recent analyses published in social-science journals using much better statistical methods and more recent data — but which do not reach Chou and Feagin’s desired conclusion.
This sort of cherry-picking is common in the literature. Another instance is provided by the historian Ellen D. Wu, who asserts in The Color of Success (Princeton University Press, 2015) that “the statistics” showing high income among Asian Americans are “misleading.” She claims that the household incomes for Asian Americans are higher because they have more workers per household and live in high cost-of-living areas, but Asian Americans supposedly receive lower returns on schooling. As evidence for these assertions, Wu cites a report that offers basic descriptive statistics on the demography and education of Asian Americans. There is no multivariate analysis of earnings or income data anywhere in the report. In other words, the evidence that Wu cites doesn’t even attempt to control for anything. Indeed, there are very few income statistics anywhere in that report. Nevertheless, this is the study Wu treats as the definitive analysis of Asian American incomes, supposedly proving her conclusion that “the statistics” are “misleading.”
These kinds of distortions and tricks are common in work on the causes of socioeconomic disparities. According to the now sacrosanct vision of stratification in American society, Asian Americans face endemic social- and labor-market discrimination because they are deemed to be “people of color” in a society organized to enforce “white privilege.” On this view, all references to the “model minority” model — which attempts to explain high rates of income and achievement among Asian Americans by reference to Asian cultural effects — are false, a mere myth. But the major tenets of the Model-Minority-as-Myth argument are themselves basically false.
In comparison to whites these days, Asian Americans are doing quite well, at least on average. A groundbreaking study found that they apparently have the highest level of intergenerational upward income mobility ever recorded anywhere in the world. With some exceptions, Asian Americans are less likely to drop out of high school, be incarcerated, or suffer poverty. They are more likely to go to college, attend prestigious universities, have higher occupational status, work in lucrative technical and STEM fields, have higher average incomes, and enjoy longer life expectancies and better health in comparison to whites.
Suppose a researcher wants to argue that having an Asian immigrant parent has no effect on the probability that a youth attends college because the researcher believes only “socioeconomic factors” matter, not Asian cultural effects. The savvy sociologist can include high-school GPA as a control variable, after which the estimated effect of having an Asian immigrant parent will be reduced. That’s overcontrolling, because one mechanism by which Asian immigrant parents have a positive effect is by enforcing higher standards about studying and homework time, thereby pushing their children to a higher GPA. But this family mechanism is effectively deleted from the estimated Asian coefficient because high-school GPA has been included as a control variable.
When social scientists try to nail down a neat model of causation, they often lose sight of the complexities and realities of human life.
Or consider the related claim, energetically promoted by Jennifer Lee, a prominent sociologist at Columbia University, that Asians face a “bamboo ceiling.” Lee has written opinion articles for the Los Angeles Times and other outlets arguing that, based on her research, Asian Americans suffer a systemic racial penalty in the job market. The problem with her conclusion is that her published statistical findings actually do not show that at all. As one of us has detailed in a peer-reviewed article, her results indicate parity with whites, except in the case of Chinese Americans, who her findings indicate are actually advantaged over whites. When a political scientist pointed out the statistical error, Jennifer Lee doubled down on her claim.
When it comes to managerial positions, there is no significant evidence that native-born Asian Americans have lower chances than whites in recent years (despite all the rhetoric about the “bamboo ceiling”); indeed, in some recent data, native-born Asian Americans may even be slightly advantaged over whites. So to portray Asian Americans as being substantially socioeconomically disadvantaged due to “white privilege” requires quite a bit of flawed research.
Lee and her co-authors are also the source of a clearly misleading but widely cited statistic: that 69 percent of Asian Americans support affirmative action. In addition to the vaguely worded and confusing survey question, the survey sample is not random or representative, favoring liberal-leaning populations.
Such ideologically driven abuse of statistics happens all across the social sciences. Why? In left-leaning academic discourse, there are strong biases toward “structural” causes, in part because scholars face strong pressures to avoid “blaming” people and cultures for social problems. But social theory must recognize both structure and agency, alongside intermediary forces of social influence such as culture.
Instead, through our theories, we tell stories. Academics, like people in general, are drawn to narratives. Much of higher ed is devoted to explaining inequality by highlighting malevolent “structural” forces. The best theories, however, must be more comprehensive. In general, social-class influences are stronger than racial influences on education, income, and health, but strong cultural patterns have emerged in some contexts — such as among second-generation Asian Americans — where cultural pressures promote behaviors that lead to high educational and career outcomes.
Again, we are not talking about normal differences in the interpretation of results. We are talking about clear errors, or at least very poor scholarship that should not have passed peer review. It is easy to question some of these results because they often don’t make intuitive sense. In the sensitive area of racial disparities, simplistic monocausal explanations like “structure” should be avoided in favor of multicausal arguments that involve the mutual influence of structure, culture, and individual variation.
People are diverse and will act for different reasons, even given similar structural environments. One of us is a cultural anthropologist who has seen this kind of diversity all over the world, where people are influenced by the cultural patterns they have grown up with. These are highly intermixed with structural environments, but neither is reducible to the other. They mutually interact and, in combination with the quirks of individual variation, create diverse outcomes. Research simply shouldn’t be directed by a priori ideological commitments. It should follow the evidence. Often, that evidence won’t lead to clear-cut or definitive results.
Some of these articles should be candidates for retraction, but retraction is rare. Despite Levy-Eichel and Scheinerman’s takedown of the book on racial disparities in postwar fiction, no corrections or retractions have been issued. Some scholars even received major promotions, perhaps partly because their findings fit favored narratives. Instead, papers that violate ideological beliefs, more than those with errors of fact, receive pressure for cancellation, often from Twitter activists.
These are not just isolated incidents. They are abetted by the entire system of research, funding, publication, and promotion. At the moment, academe strongly values findings of racial discrimination. The rewards for such findings are therefore high, and people will go to extraordinary lengths to achieve them.
The people who commit such errors sense that reviewers and editors will not check them on it. That fact helps explain some of the attacks on academe from the right. “Progressive activists,” only 8 percent of the population, now dominate much of the social sciences and humanities. There should be a way to check institutions whose groupthink produces flawed research, though that of course has its dangers. The activists’ enemies on the hard right control many state legislatures and, as in Florida, are attempting to legislate speech in higher ed. This is a recipe for continued polarization and conflict, not for truth.
Because many academics have limited knowledge of statistical models and their weaknesses, it is easy for people to be misled by studies that can’t possibly show what they intend to show. The studies can be manipulated through various statistical techniques, either intentionally or unintentionally. More important, models often can’t reflect the complexity of causation, which often involves feedback loops and lagged effects that make causation extremely hard to determine. One example is the debate over “broken windows” policing, where, despite many statistical studies over the years, there is no consensus on whether it or something else (like social control) reduces crime.
Any time a study comes up with largely “structural” reasons for disparities, be suspicious — just as suspicious as you would be of a study with exclusively individual-level causes. When social scientists try to nail down a neat model of causation, they often lose sight of the complexities and realities of human life.
