The University of Illinois at Chicago recently found itself living a modern nightmare: Google’s automated cybersecurity regime mistook the university as the culprit in a spam attack on the university’s students and began blocking university email accounts from sending messages to Gmail users.
The blocking went on for more than two weeks, and the affected Gmail users included 13,000 of the university’s own students. University officials describe those two weeks as a Kafkaesque state of limbo.
On March 27, 12 days after Google blacklisted the university’s domain, university officials wrote on an academic computing blog: “We have followed the instructions they have posted online on how to resolve this issue, but there is no indication on what happens once they receive the request.”
“Experience from other sites that have encountered this issue range from days to weeks,” the officials wrote. “The only recourse is to wait until they stop blacklisting uic.edu, which happens on its own, but not on a set schedule.”
Google officials told The Chronicle on Monday that the issue had now been resolved. But the university’s two-week struggle highlights the hazards that come with relying on an outside company—especially one that depends heavily on automated processes—to deliver messages to students.
The email interruption may have wrought havoc on the campus. Last week one medical student asked for help on a Google Products Forum.
“This has severely affected all members of the school,” the student wrote. “This school is the major public university in the Chicagoland area and includes medical professions along with undergraduate studies. Missing emails has made people miss important deadlines, meetings, and, in some cases, may have damaged career opportunities.”
One user responded to the student’s plea, but offered no quick fix. “There is nothing else to but be patient,” the user wrote.
It Started With a Spam Attack
The email crisis began on March 13, when spammers began attacking the university’s email system.
Spam attacks are common at universities and other organizations that have the power to relay messages to many accounts. At Indiana University, more than 90 percent of the email sent to university accounts is spam, according to Dennis J. Cromwell, Indiana’s associate vice president for enterprise infrastructure.
Indiana subscribes to a “reputation filtering” service that automatically blocks messages sent from known bad actors before they get into the university’s email system. As a result, most messages are “turned away at the border,” said Mr. Cromwell.
At the University of Illinois at Chicago, however, the spam from the March 13 attack got through. “Until the antispam system we use detected and caught this increase in spam, the spam was delivered to our users,” the March 27 blog post said. The spam continued to get through until March 17, according to the post.
The Chronicle tried contacting Cynthia E. Herrera Lindstrom, the university’s chief information officer, to ask how the spam breached the university’s system. She had not responded to inquiries as of Monday evening.
So how did the university end up being marked as the perpetrator of the March 13 attack, rather than the victim?
Like many universities, Illinois at Chicago uses Google Apps for Education, which offers a free, Gmail-based service to colleges. But the university also gives students the option of having emails routed to a personal account.
About 13,000 students—almost half of those enrolled at the institution—have their university email routed to a personal Gmail account, according to William Allen Randall, director of enterprise architecture and development.
When the university began routing the spam emails to 13,000 private Gmail accounts, the spam-detection software at Google marked the “uic.edu” domain as the source of the potentially harmful messages.
‘No Number You Can Call’
On March 15, two days after the spam attack began, the company began to block emails sent from the university’s domain to any Gmail accounts.
Not all the emails were blocked, said Mr. Randall. Google put a “rate limit” on emails sent from the university’s domain, he said, essentially putting a cap on how many emails could be sent per day from university accounts to external Gmail addresses. Some messages made it to their intended recipients, but many others were delayed indefinitely.
University officials were at a loss as to how to fix the problem. “There’s no number you can call or people you can reach,” Mr. Randall said in an interview on Monday.
The university tried getting help through its contacts at Google Apps for Education but was told that Google Apps and Gmail were “different entities” and “don’t have that sort of relationship,” he said.
University officials updated their March 27 blog post late Monday afternoon to say that the issue had been resolved, 16 days after Google blacklisted the university’s domain.
“To prevent similar issues from happening in the future,” the officials said, “we are working with IT Governance to analyze what happened and are looking at possible changes.”
