Supercomputers, a Status Symbol in Academe, Compete With the Cloud

Even with one of the nation’s biggest academic supercomputers right on her campus, Clemson University’s Amy W. Apon wanted more speed.

The computer-science professor has now found it with a fairly utilitarian choice: Amazon Web Services.

In a test just now concluding, Ms. Apon led a team that ran 17 years’ worth of abstracts from academic journals through the popular cloud service to try out some machine-learning software. For comparable equipment-usage costs, she says, Amazon appears to have run the test about 40 times as fast as Clemson’s Palmetto2, the nation’s eighth-most-powerful academic computer.

Palmetto2 is still a computational marvel. But the “speedup” with Amazon, Ms. Apon said, “comes from using a lot more whole computers than is available on Palmetto.”

For a world in which having a great big supercomputer on campus has become a badge of big-league status in academe, a shift seems underway. On-site computing capability does remain important. But with demand soaring across academic fields, budgets still tight and commercial cloud options growing, value is increasingly attached to having in-house computer experts who can guide scientists to their best options in processing resources, either on campus or off.

Examples of the shift include Imagine Rio, a web-based resource for historians at Rice University that depicts the street-level growth of Brazil’s Rio de Janeiro year by year back to the city’s founding, in the 1500s. Rice doesn’t have especially large computing resources — less than a tenth of what can be found 200 miles away, at the University of Texas at Austin, home to the nation’s two biggest academic supercomputers. But Rice does have the Ken Kennedy Institute for Information Technology, which helps about 170 faculty members with advanced computing needs.

One of them, Farès el-Dahdah, a professor of humanities, had the concept for Imagine Rio but little idea of how to pull it off. At the institute, his team was guided from developmental work on Rice computers to Imagine Rio’s eventual hosting on the Amazon cloud. The lack of a supercomputer at Rice made “zero difference,” he says.

That perspective differs at supercomputer giants such as the University of Texas and the University of Illinois at Urbana-Champaign. They’re among several institutions that promote their top-rated machines — many financed with federal grant support — as crucial tools and important lures for top researchers and students.

The University of Texas’ president, Gregory L. Fenves, marked the operational start in July of Stampede2 as the most powerful supercomputer at any U.S. university, as determined by the TOP500 project, a widely recognized measure that ranks computers by their ability to solve a particular set of linear equations. The machine, he said, would let scientists at Austin “take on the greatest challenges facing society.”

Return on Investment

Various financial estimates, by universities and others, have backed the wisdom of such investments. Institutions seeking government money for the most powerful tier of computing do need to build advanced hosting facilities with sophisticated cooling systems. Texas spent $38 million expanding its data center before winning more than $100 million in support from the National Science Foundation for its two Stampede systems.

But when it comes to return on investment, a study last year found that universities realized more than $40 for every dollar invested in high-performance computing. The University of Illinois issued a report this year saying its Blue Waters supercomputer, for an investment of some $500 million, mostly from the NSF, will result in a $1.3 billion economic gain for the state.

And the demand for supercomputing among researchers looks strong. Texas’s previous supercomputer, the first Stampede, supported some 3,700 projects over four years in almost every academic field, says Dan Stanzione, executive director of the Texas Advanced Computing Center. Over 15 years, the investment has helped attract about $1.5 billion in research grants to campus, he says.

Because such systems are built largely with federal money, the university has direct control over only a small share of usage time on the NSF-funded portions of its systems. Most of that time is allocated nationwide through a competitive system. And, as with federal grant applications, demand for advanced computing well exceeds supply.

For Stampede’s systems, Mr. Stanzione says, researchers request five to seven times more time than is available. Actual interest is probably much higher still, he says, because supply shortages lead applicants to scale down their ambitions.

Martin Berzins knows that. After a truck carrying 8,000 wrapped explosives used in seismic testing overturned in 2005 about an hour outside Salt Lake City, exploding and wiping out the roadway beneath it, Mr. Berzins, a computer-science professor at the University of Utah, led a team that spent five years on computer simulations to figure out what had triggered the detonation and how best to safely pack such cargo. He also has worked with General Electric on improving the design of its coal boilers, using supercomputers to model machines the size of the Statue of Liberty to single-millimeter detail.

Utah is not quite in the top tier of supercomputer institutions, but Mr. Berzins — who has used about a billion processor hours in the past three years at the NSF-funded facilities in Texas — is grateful that some U.S. universities are. Even companies that might benefit from safer explosives and cleaner power systems might not willingly bear the research costs, he says, adding that paying too much attention to university-specific cost-effectiveness measures “could be tragic.”

Hard data on supercomputer costs and benefits are elusive. Partly that’s because benefits, such as institutional reputation and attractiveness to faculty members and students, can be intangible. And partly it’s because important costs, like power usage and the profusion of individual components, are often poorly tracked.

“Universities are lousy at counting total cost of ownership for operating almost anything,” says Jan E. Odegard, executive director of Rice’s Ken Kennedy Institute. “A university can pretty much make any arguments it wants” about supercomputer cost-benefit ratios.

Ms. Apon, of Clemson, published a study in 2015 comparing universities with a top-500 supercomputer and those without one, which Clemson proclaimed as long-awaited proof of the value of such an investment. But the paper correlated supercomputers largely with increased research output; key costs, it acknowledged, were “hard to quantify.”

An author of the study about return on investment, Steve Conway, says the importance of a particular university’s on-campus computing capacity is often overstated. He endorses an emphasis on providing faculty members with computing expertise, and on deeply integrating computing skills into the curriculum.

Fewer than 10 percent of U.S. research universities are properly preparing their engineering and science majors with necessary competence in high-performance computing, says Mr. Conway, now a research vice president in the High Performance Computing Group at Hyperion Research, a technology consultancy.

That concern was reflected in an analysis published last year by the National Academies and funded by the NSF. It urged robust federal spending on supercomputing as part of a national strategy of economic competitiveness. The report emphasized the need for human ability as much as for machinery. Hardware, software, computing services, and expertise should be “considered in an integrated manner,” it said.

China, as with many other areas involving science and economics, has drawn particular attention in supercomputing. It now has 160 of the world’s 500 top-ranked machines, only eight fewer than the United States. Some of China’s initial effort may have been about prestige, but such numbers now show a strategic move, says Mr. Stanzione, of the Texas Advanced Computing Center. The availability of advanced computing is widely understood to be “the rate-limiter on innovation,” he says.

By academic field, major users of supercomputers include genomics, robotics, neuroscience, chemistry, civil engineering, physics, evolutionary biology, advanced materials, drug discovery, and artificial intelligence, and machine learning. Beyond the National Science Foundation, another big federal spender on advanced computing is the Department of Energy, in both its science and weapons divisions. And as Mr. el-Dahdah’s project at Rice shows, the humanities and social sciences are showing increased interest.

That’s forcing service providers to adjust. For the Clemson study compiling journal abstracts, Ms. Apon developed software manipulations to improve the efficiency of lower-priority computations that generally await lulls in supercomputer activity. For many users, says Rice’s Mr. Odegard, such programming techniques are whittling away the relative advantages of elite-level computer hardware.

The Human Factor

At the same time, he says, human factors can hinder such efficiencies. Researchers in many fields are accustomed to having on-demand computing capabilities — their personal laptops or desktop systems — and don’t grasp the necessity of waiting a day or two for results when they begin using supercomputers. “Some folks have zero tolerance for any delay,” even if it benefits the larger community, Mr. Odegard says. “It’s almost more of a social problem right now. But it’s social on both sides of the island — we, as operators of the infrastructure, also have to learn and listen to them.”

It’s one of many ways that universities are seeking the right balance with supercomputers. Those bidding for such systems are typically state institutions, says one supercomputer analyst, Thomas R. Furlani, of the University at Buffalo. That’s because the main sources of support for ancillary costs, like infrastructure and power, are allocations of tens of millions of dollars by state lawmakers who accept assertions about the job-creating potential of such machines. “The university cannot do it on their own — it’s a losing-sum game for them,” says Mr. Furlani, director of Buffalo’s Center for Computational Research.

The University of Texas decided to become a major player in hardware in the early 2000s. Cornell University and Princeton University were part of the NSF supercomputer program at its founding, in the early 1980s, but both later got out. (Neither has any comment on the decision, with one of their computing directors calling it a sensitive topic.) Both remain members of the Coalition for Academic Scientific Computation, which comprises 84 U.S. institutions pursuing advanced computing technology.

From that group, a new $120-million grant that the NSF offered in July for its next supercomputer project is likely to have only a handful of applicants when the bids are opened in late November.

Even the University of Illinois, patriarch of academic supercomputing, has heard doubts about its ability to persist in the face of its state’s extensive budget problems. “I won’t say it hasn’t been a challenge,” says William D. Gropp, a professor of computer science and director of the university’s National Center for Supercomputing Applications. “But we will be putting in a strong proposal.”

“It’s a high-stakes game,” says Mr. Stanzione of Texas. “A certain number decide to play.”

Supercomputing: The Big 20
Here are the 20 most powerful campus-based supercomputers in the U.S., according to the TOP500 project, which ranks systems based on their performance in solving a set of linear equations.

Source: TOP500 project (https://www.top500.org/statistics/sublist/)

Paul Basken covers university research and its intersection with government policy. He can be found on Twitter @pbasken, or reached by email at paul.basken@chronicle.com.

Clarification (11/1/2017, 10:48 p.m.): This article has been updated to clarify an explanation of the speed of Amazon Web Services relative to Clemson University’s Palmetto2 supercomputer.