Dizzied by Data

In his short story “The Library of Babel,” Jorge Luis Borges imagined an infinitely large library containing all books. Although the library was wondrous, people had no way of finding the right book. Much like Borges’s library, the information age has presented us with a dizzying amount of data. The past decade witnessed the rise of the interactive Internet—Web 2.0—where people not only consume information but also add to it. Millions of people started blogging; social-networking sites like Facebook amassed half a billion users; and sites like Wikipedia enticed people to collaborate and share their expertise.

To cope with all this data, we created new ways to find it and analyze it. Search engines like Google revolutionized our ability to locate information, and data-mining technologies were developed to detect patterns and make judgments about people’s interests and behavior.

Over the next decade, the ability to search for information and to analyze it will mature dramatically. According to John Battelle in his book, The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture (Portfolio, 2005), “As every engineer in the search field loves to tell you, search is at best 5 percent solved—we’re not even in the double digits of its potential.”

Data-mining technology will also continue to develop at a rapid pace. Information is being collected and disseminated at an exponential rate. It is no longer enough to shield data from prying eyes. It is already very difficult to de-identify data because so much information about people is now available that enables the data to be reidentified. For example, researchers were able to take a database of movie ratings by anonymous people and identify 80 percent of the raters based on how they rated three movies.

The growth of information-analysis technology will have profound consequences, both good and bad. An enhanced ability to search for and analyze data will facilitate research, communication, and knowledge. On the other hand, it will be harder for people to escape mistakes they made in the past. Big corporations and the government will be able to learn more about our lives and have more power as a result. The norms and legal rules we develop over the next decade to cope with these developments will determine the limits of our freedom and privacy.