Colleges no longer simply want to know what their students know, but how they think.
Higher-order thinking skills are “something that schools are paying a little bit more attention to these days,” says Jeffrey Steedle, a measurement scientist at the Council for Aid to Education, whose Collegiate Learning Assessment essays are used at several hundred colleges to test students’ abilities to synthesize arguments and write persuasively. “It’s largely in response to the recognition that these skills are needed to be competitive in the global marketplace.”
But educators also say that paper-and-pencil examinations have limits—for one thing, knowing that you are being tested can drag down performance—and they are looking for new methods to measure skills like critical thinking, creativity, and persistence.
Valerie J. Shute, an associate professor of educational psychology and learning systems at Florida State University, believes she has a solution in “stealth assessment"—the administering of tests without students’ knowing.
To do that, Ms. Shute and other stealth-assessment researchers have turned to video games, which let educators watch students solve complex tasks while immersed in virtual worlds. How students react to new challenges and put evidence together—without the pressure of test proctors breathing down their necks—can reveal a lot about creative problem-solving skills that traditional testing cannot deliver. “A lot of important stuff happens when playing games,” Ms. Shute said. “You’re just doing. You’re in the process.”
Ms. Shute, who first studied stealth assessment with a video game for undergraduate students more than 20 years ago, sees applications for students of all ages. She is now helping two of her graduate researchers test the technology with sixth graders in Florida, And she is already looking for new ways to use the technique in her graduate courses.
“Everybody likes to play,” she says. “And so much could be done using games.”
Matthew Ventura, an associate research scientist at the Educational Testing Service who worked on stealth-assessment research with Ms. Shute, says the technique will be especially helpful in eliminating test anxiety, which can hurt students’ performance. That and its potential to test intangibles like creativity and problem solving, he said, make the technique attractive to educators and video-game developers alike.
Virtual Worlds, Real Skills
Educators have long believed that video games and virtual worlds could be used to supplement classroom instruction, although not necessarily as testing tools.
In 1986, when Ms. Shute was a postdoctoral fellow at the Learning Research and Development Center at the University of Pittsburgh, she designed a computer game to teach undergraduates the principles of microeconomics. The game was set in the virtual farming community of Smithtown, where students could play around with different economic factors—like product price, labor costs, and population size—to see how they affected the market. A student who decided to raise the price of coffee, for instance, would see demand for his or her product fall as a result. The software also let students plug in hypotheses about the outcome of their experiments.
On its surface, the game was a way for students to get acquainted with basic economic ideas—and maybe have a little fun in the process.
But it was also teaching students a deeper lesson about scientific inquiry. If they changed too many inputs at once, they found it difficult to determine what caused a sudden change in the market, and had to go back and experiment with the inputs to determine exactly what was happening.
The question that grew out of Smithtown, Ms. Shute says, was, “Wouldn’t it be lovely to actually pass along the log files of what students did in order to look at their scientific-inquiry skills?”
Since then, Ms. Shute has been working on a framework to help educators design and execute stealth assessment.
She looks first to the core competencies—critical thinking, empathy, persistence—that she wants to test, then breaks them down into smaller goals. She can then tie those theoretical skills to actual tasks in a video game. For instance, an instructor looking to test a student’s grasp of systems thinking—understanding the complex relationships among parts of a whole—might ask players to complete tasks that show information gathering, developing hypotheses, and tracing causal relationships.
Rather than conducting a one-off exam, Ms. Shute says, continuing stealth assessment “enables you to have a very systematic representation of the stuff you’re interested in.”
But for Ms. Shute, stealth assessment is not just about gathering data. It’s also about improving teaching.
If instructors know where students need the most help, they can quickly tweak their courses—and their games—to make up for those deficiencies. Students who need help developing critical-thinking skills, for instance, may be asked to repeat a level or to take on additional tasks in the game until their performance is satisfactory—all without interrupting their play. “The idea of stealth assessment is really to make it merge into the fabric of the learning environment,” Ms. Shute says. “My goal is to blur the distinction between learning and assessment.”
Crunching all the data on student performance and tweaking computer games were major tasks in the days of Smithtown, but today’s technology is making it possible to test increasingly complex critical thinking using virtual worlds.
Taiga Park, a computer game developed by the University of Indiana’s Center for Research on Learning & Technology, is one of Ms. Shute’s favorite vehicles for stealth assessment. On its surface a game about ecology, Taiga Park requires players to look for the cause of a widespread fish die-off in a virtual river by “interviewing” park rangers, environmental scientists, and the owners of a logging company. While students learn about pH levels and runoff, they also come away with lessons on data analysis, complex cause-and-effect relationships, and communication.
With the stealth-assessment framework behind it, Oktay Donmez and Yoon Jeon Kim, two of Ms. Shute’s graduate students at Florida State, are planning to test Taiga Park with about 50 sixth graders. Starting in the spring, the researchers will study the students’ performance data from the game to see how effective stealth assessment is at measuring their complex thinking.
“My goal is not only to design, develop, and implement stealth assessment within a game, but also to test its effectiveness,” Mr. Donmez says.
Although much of the research in the field—especially when dealing with elementary and secondary students—is centered on video games, Ms. Shute is quick to point out that stealth assessment is widely applicable as a tool, not just a game.
Working with Florida State graduate students in a course called “How to Write Excellent Literature Reviews,” Ms. Shute found that she could use routine assignments—like peer reviews and summaries of research material—to analyze her students’ higher-order thinking skills. All assignments can be linked back to a larger skill, she says. “Evidence is everywhere.”
Ms. Shute also hopes that stealth assessment might engage students unmoved by traditional teaching and testing.
“We have this whole group of kids who are not engaged with school, and appropriately so, because schools are so antiquated,” she says.
With field research on stealth assessment still ahead of them, both Ms. Shute and her students are optimistic about the technology, especially for a generation of students who grew up with video games at home.
“They’re going to play video games anyway,” Ms. Kim says. “We can actually embed something that can help them learn.”
