Testing Companies Try to Balance Secrecy and Openness

At first there were just gripes. Moments after taking the revamped SAT last Saturday, high-school students throughout the nation shared their reactions online. “New SAT is HORRIBLE,” @TSpillaneUSA tweeted. “Umm … So that sucked,” @_shai7 wrote. These were exhausted teenagers, blowing off steam.

But by 1 p.m., Eastern time, some students were doing more than that. They were recalling specific questions, some in considerable detail. On College Confidential’s anonymous discussion forum, Colai2zo asked about a mathematics problem: “Did anyone get the really long polynomial function one?” Four minutes later, premednyy replied: “You had to add the exponents to see which ones equal 15, then multiply the coefficients and then combine them. The answer as i recall, was 52.”

As similar exchanges continued into the evening, the College Board, which owns the SAT, and the Educational Testing Service, which administers it, monitored various websites, looking for posts that referred to specific test questions, ready to pounce, if necessary. It’s one way of protecting an examination’s integrity from would-be cheaters, the just plain curious, and anyone else who wants a peek.

Keeping the contents of high-stakes tests under wraps is an increasingly sophisticated task in an age of free-flowing information. Yet tension between secrecy and transparency has long defined the testing industry. Although the public wants fairness — safeguards to ensure that nobody can get the answers in advance — fee-paying families also expect openness from the organizations behind all of those questions.

They haven’t always gotten the latter. These days, the College Board makes free sample tests and practice materials available. For much of the 20th century, though, the nation’s most prominent tests were largely hidden from view, and those who created and sold them wanted it that way.

During the late 1970s, the testing industry fought a New York State bill allowing test takers to obtain copies of their admission exams along with their answer sheets and an answer key. The College Board and other testing companies argued that the practice would increase costs because once an exam was disclosed, it could not be reused. The need for more questions, they said, could reduce test quality.

Principled snootiness informed those concerns, as a monograph about the debate reveals. “This law cannot force test companies to explain the meaning of test scores to students,” one opponent wrote in 1979. “The construction, evaluation, and interpretation of tests are highly technical matters which must be dealt with by … those who are trained in this speciality.”

One New York legislator saw the issue much differently: “The testing industry, despite its influence over vital aspects of individuals’ lives, operates in a most unaccountable fashion. … Almost totally exempt from public scrutiny.”

New York’s “truth in testing” law was adopted by the State Legislature in 1979. Two years later, the College Board decided to let test takers outside the state request copies of exams upon receiving their scores, effectively making the law a national policy. In a 1981 New York Times article, one of the organization’s trustees described the move as a response to “changing conditions and to a consumer point of view.”

Nowadays the College Board’s Question-and-Answer Service represents a further compromise with consumers. It allows test takers to order (for $18) question booklets and answer sheets from three of six annual testing dates in the United States (ACT Inc. provides a similar service, at $20, also for three of six test administrations).

In short, some versions of tests are destined for immediate public consumption, and others are not. That might help explain why the College Board, in an unprecedented move, barred nonstudents from taking the SAT last week. Because the organization does not plan to make that edition available right now, only high-school students got an early look at the retooled exam, which apparently will be reused at a later date.

The College Board’s vague explanation for the switch — concerns about test “security” — baffled test-preparation companies, which have long sent employees to take the test each time it’s offered.

In an email announcing the move, the College Board said it wanted to keep out those not taking the exam “for its intended purpose.”

That rekindled an old question: Who has a right to see a test?

‘It’s a Chess Match’

The answer is complicated by the constant threat of cheating. Testing companies must now contend with tiny recording devices and gadgets that can transmit instantaneous messages across the globe, not to mention people stealing test booklets.

“It’s a chess match, and we always want to be one move ahead,” says Ray Nicosia, director of testing integrity at the Educational Testing Service, where he oversees a staff of 70. He has personally removed camera watches from the wrists of test takers capturing images of exams.

Over the years, ETS has beefed up its considerable security measures. Following a recent cheating scandal in New York, where students paid others to take the SAT for them, registrants must now upload a photograph of themselves when they register online. On testing days, they must present a driver’s license or passport that matches the face on their printed ticket (and the one that proctors see before them).

The rule has reduced incidents of impersonation, according to ETS. That just means cheaters look for other strategies. “When you close one door,” Mr. Nicosia says, “they’re gonna try to kick down another.” Like communicating during tests.

Although ETS does not search SAT takers for cellphones or digital devices, they are told that proctors who see or hear one will dismiss them, resulting in canceled scores.

Once the test is over, ETS and the College Board, like ACT Inc., scour the Internet to see what students are saying. And they say a lot.

Anyone who registers for the SAT agrees to abide by terms and conditions that prohibit the sharing of test questions or answers, and discussions of the test by any means: “There is never any point in time at which you are allowed to discuss exam content unless it is released as part of a College Board service.”

Yet some students do it anyway. When testing organizations spot a clear-cut copyright violation (the rare case, say, when someone gets hold of questions and posts them verbatim), they demand an immediate deletion. Much of what students say online, however, is murky.

College Confidential warns users against discussing the content of tests. “We don’t want to be a cheating site,” says Roger Dooley, a co-founder of the ever-popular discussion portal. Moderators can delete posts they deem too explicit.

Still, there’s judgment involved. “It’s always a gray area — it really has to do with how specific it is,” Mr. Dooley says. “Most of it is relatively innocent, students trying to figure out if they got the right answer or wrong answer.”

Some online conversations get too specific for the testing industry’s liking. Hours after taking the SAT last Saturday, dozens of students shared details about the test on Reddit, an anonymous website. Minute by minute, they compared notes on questions, like one on the math section that asked about the odds of getting a chair in the sixth row, and whether the word “open” in one reading passage meant (a) vacant or (b) uncovered.

Then someone posted a link to a Google Doc (“March 5 New SAT”) in which at least a dozen users tried to reconstruct the exam they had just taken — all of the questions and each of the answer choices — from memory. Participants typed furiously in various colors, editing one another’s additions. After an hour or so, the file grew to 64 pages, a partial, crowdsourced glimpse of the SAT.

Then someone pasted the College Board’s policy on discussing exam content into the document. “Wee-woo-wee-woo that’s the sound of college board police,” one user wrote at 4:25 p.m.

A Reddit moderator later posted a warning: Someone claiming to represent a “security agency” hired by the College Board had contacted moderators, asking them to remove comments about test questions. “The agency may attempt to determine your identity,” the message said, “based on posts made in that discussion thread and elsewhere.”

The College Board declined to discuss the matter. An ETS official said only that the College Board had deployed “take-down actions.”

Although some Reddit users have since deleted their messages, the moderator was defiant: “We will not acquiesce to this agency’s demands.”

Activity has ceased in the Google Doc, where the text, now crossed out, is difficult to read. But one can still make out the words of one participant, who seemed to express what the anonymous team of SAT sleuths had been thinking.

In response to the reminder about not discussing the SAT, the student wrote: “We all took it already.” Now that the test was over, it was theirs.

Eric Hoover writes about admissions trends, enrollment-management challenges, and the meaning of Animal House, among other issues. He’s on Twitter @erichoov, and his email address is eric.hoover@chronicle.com.