Report: “Facts Are More Important Than Novelty: Replication in the Education Sciences”

Authors: Matthew C. Makel, gifted education reseach specialist at the Duke University Talent Identification Program, and Jonathan A. Plucker, a professor of educational leadership at the University of Connecticut

Summary: The authors of the study, published in Educational Researcher, the journal of the American Educational Research Association, set out to learn how often studies in the field get formally replicated. They compiled a list of the top 100 education journals, based on how much the journals were cited over five years, and calculated the share of articles ever published in each that involved attempts to replicate earlier research. Among their findings:

• Of the articles published in top education research journals, only 0.13 percent, or just over one in every thousand, were based on replication studies.
• Replication studies accounted for an even smaller share of articles posted by most journals dealing with higher-education research. The exceptions included the Journal of American College Health, Research in Higher Education, and the Journal of Diversity in Higher Education, which stood out as having more than 5 percent of its articles devoted to replications of previous studies.
• Nearly half of all replications of education research were published by the same research team that conducted the original research. Whether an author of the original study was involved was strongly correlated with the likelihood of a study being successfully replicated, a finding that raises questions about bias. Attempted replications involving none of the original authors were successful 54 percent of the time, whereas those involving at least one of the original authors were successful about 89 percent of the time when published in the same journal as the original study and just over 70 percent of the time when published in a different journal.
• Although replications of education research remain rare, their numbers appear to be increasing. Currently, about one in 500 studies are replications, up from one in 2,000 in 1990.

Bottom Line: Like many social sciences, education suffers from a dearth of research validating the results of earlier studies, creating the potential for error and fraud. Among the factors blamed: Journal publishers appear predisposed to reject replication studies in favor of new research, potential employers can be less impressed with replications than original work, and researchers can see little incentive to conduct replication studies as a result of the lack of interest in them.