Members of the Association for the Study of Higher Education this week heard a sharp critique hurled at the influential annual study that many have relied on or used as a model for their own work: the National Survey of Student Engagement, widely known as “Nessie” after the acronym NSSE.
In a paper presented at the group’s annual conference here on Friday, just three days before the scheduled release of this year’s NSSE results, Stephen R. Porter, an associate professor of research and evaluation at Iowa State University, argued that the survey of undergraduates “has very limited validity for its intended purposes and that researchers and institutions must adopt a new approach to surveying college students.”
Tapping into a large body of other research dealing with how people respond to survey questions, Mr. Porter complained that NSSE asks many questions that are of dubious relevance, are too vague for the answers offered by students to be meaningful, or fail to take into account shortcomings in human memory and the difficulties involved in precisely measuring attitudes.
In an interview, Alexander C. McCormick, director of NSSE, challenged some specific criticisms contained in Mr. Porter’s paper and argued that NSSE administrators have determined through extensive discussions with student focus groups that students have very similar interpretations of the survey’s questions. He added, however, that NSSE administrators are well aware the survey has some flaws that are likely to result in errors in some of its measures of students.
“Any survey instrument is a blunt instrument,” said Mr. McCormick, an associate professor of education at Indiana University at Bloomington. “I think there is a lot in this paper that will be helpful to us as we think how we can improve NSSE.”
Mr. Porter told his audience at the conference that he had chosen to “take a bold stand” in criticizing the survey because it plays such a major role in influencing college operations, government policy, and students’ decisions about where to enroll. He said the annual reports of the survey’s findings have “potentially life-altering consequences” and quite possibly have caused some colleges to be unfairly regarded as poor environments for students.
Mr. Porter stressed, however, that while his paper focused on NSSE, his intent was to make the broader point that many education researchers are surveying students with questions of dubious validity, which ask students to provide assessments of their attitudes or factual information about their behavior that many are unlikely to report accurately.
He said other social sciences have much more rigid criteria for judging whether survey questions are valid, and he argued that many education researchers are under such pressure to produce publishable studies that there is little incentive for them to take the time necessary to test whether their questions are valid.
Varying Interpretation, Vague Quantifiers
Other research presented during the same session similarly raised questions about the reliability of student surveys. For example, Linda DeAngelo, assistant director of research at the Cooperative Institutional Research Program at the University of California at Los Angeles, and Serge Tran, associate director of data management and analysis at the university’s Higher Education Research Institute, presented study findings showing that, while most students report their SAT scores fairly accurately on surveys, low-scoring students are more likely to be off in the numbers they give, and high-scoring students are more likely to exaggerate how well they did.
“Have we arrived at a point where all is not what it seems from the data we collected?” asked the session’s moderator, Nathaniel Bray, an assistant professor of higher education at the University of Alabama at Tuscaloosa. Noting how education research is held to different standards than research in other fields, he said, “This is a debate that has been coming for a long time.”
One of Mr. Porter’s chief criticisms of NSSE is that many of its questions use words that are open to varying interpretations or ask students to report the frequency of behaviors on scales using vague quantifiers, such as “often,” rather than actual numbers. As a result, his paper says, “it is likely that students do not understand much of what we ask them.”
The 2009 survey, for example, asked students how often they had “discussed grades or assignments with an instructor” without clarifying whether “instructor” referred only to faculty members or also included graduate students who teach. Similarly, the survey asked students how often they had “serious conversations” with peers about certain subjects without clearly defining what was meant by either word, or posed questions using educational jargon, such as the phrase “thinking critically,” that probably went over students’ heads.
Mr. McCormick, the survey’s director, acknowledged that NSSE is somewhat inconsistent in using the terms “faculty member” and “instructor” but found in its focus groups that students tended to use both terms interchangeably. He said NSSE generally could tell if students were having difficulty understanding a question — and make appropriate revisions — by looking at whether a large share of students had skipped over it.
In criticizing the survey’s use of vague quantifiers, Mr. Porter’s paper cited other research showing that students have very different ideas of what terms like “very often” mean in respect of various behaviors. When asked separately to provide actual numerical estimates of how frequently they engaged in certain behaviors, some who previously had answered “very often” might say once a week; others, a dozen times, or more frequently than that.
If students are confused about the meaning of a question, they often take cues from the scale of answers they may give — thinking, for example, that because they see themselves as average, they must engage in an activity sometimes — or provide what they consider the “right” answer, reasoning that because they are good students and good students probably engage in a specified activity often, they must engage in it often as well.
‘Computer Hard Drives in Their Head’?
Mr. McCormick argued, however, that NSSE determined in its focus groups that students typically meant about the same thing when they offered a response like “often” to a given question. And he said Mr. Porter’s critique appeared to assume mistakenly that NSSE seeks to quantify, in precise numbers, how often students engage in certain activities, when its intent is mainly to make relative comparisons between different subsets of students or different institutions.
Another major criticism offered by Mr. Porter’s paper was that NSSE and similar surveys “implicitly view college students as having computer hard drives in their head” and thus being able to accurately recall their actions weeks, months, or even years before. Memories fade over time, and, because people have difficulty assigning dates to past events, they sometimes recall things that happened some time ago as having occurred more recently than actually was the case. As a result of those two flaws in memory, college seniors may report having engaged in certain activities more often than freshmen simply because they have more college memories to draw upon, and their memories of engaging in those activities in their junior or sophomore year are bleeding into their recollections of more recent events.
The survey’s questions about attitudes are especially fraught, Mr. Porter’s paper says, because they appear to assume “attitudes exist in the respondent’s head, and all the respondent has to do is reach in, read the file, and report an answer.” In reality, he said, research shows that “most attitudes are rarely formed until a respondent reads the question, and attitudes vary greatly over time, due to respondents’ forming and reforming an attitude each time they are asked a question about the attitude.”
Mr. McCormick agreed that “there are a lot of problems with attitude questions,” and he said “that is why we don’t have many on the survey,” which focuses on students’ behaviors.
Broadly, Mr. McCormick rebutted Mr. Porter’s critique as not fully taking into account students’ survey-taking behavior and the practical need to gather information from large numbers of them. Earlier versions of NSSE, which was first administered in 2000, contained much more elaborate instructions, but students later admitted skipping over them.
Any survey administered to students, he said, needs to strike a balance between clarity in its instructions and the wording of its questions and “actually being something students are going to respond to.” NSSE administers conceivably could construct a survey in which every word was carefully defined, “but if hardly any students are going to fill out that survey, then that effort is wasted.”
Mr. Porter’s paper suggests that, instead of trying to measure student engagement through survey questions asking them how often they engage in certain activities, education researchers should borrow from other fields, such as economics, and ask students to keep daily diaries in which their accounts of how they spend their time will be less prone to memory lapses and thus more accurate.
The paper says such changes “will require a serious reorientation of how we study students,” in part because high costs will be involved in paying students to keep diaries and converting the information they provide into usable data sets. But, it argues, the payoff in terms of accuracy will be worth the costs.
