Follow Datanami:
August 27, 2012

The NSA, Big Data, and “Total Information Awareness”

Datanami Staff

Big data research is generally optimistic in its aims. From advancing personalized medicine or predicting airplane failure, many in the big data industry are seeing an ever-growing bevy of useful, helpful applications on the horizon.

However, as Shane Harris, author of The Watchers: The Rise of America’s Surveillance State, reports, the National Security Agency is using this research for more sinister purposes.

According to Harris, the program was conceived when former Reagan adviser John M. Poindexter outlined something called the T.I.A., or Total Information Awareness, in March of 2002. The program’s aim was to monitor America’s digital communications in an attempt to seek out terrorist plots. Even within only six months of 9/11, Poindexter’s plan was criticized for its obvious invasion of privacy and eventually scrapped.

However, apparently the NSA took the outline of this plan, made it more invasive, and instituted it themselves. “A decade later,” Harris writes “the legacy of T.I.A. is quietly thriving at the NSA. It is more pervasive than most people think, and it operates with little accountability or restraint.” According to Harris, the NSA took the initial TIA program and eliminated two constraints: making data anonymous such that it could only be linked to a person through a court order, and a set of logs whose job it was to determine if certain citizens were being unfairly targeted.

How big data fits into all of this is simple: without the ability to store and process all of America’s communications, the NSA’s system would be pointless. According to Harris, the NSA is going to ridiculous lengths not only to collect the data but also in storing it. “Today, this global surveillance system continues to grow. It now collects so much digital detritus — e-mails, calls, text messages, cellphone location data and a catalog of computer viruses — that the N.S.A. is building a 1-million-square-foot facility in the Utah desert to store and process it.”

Further, if there exists enough data to build a million-square foot facility to house, there is certainly too much to analyze it through standard means. Complicated analytics have to be applied to the data before humans can even make sense of it, much less use it. Harris believes the NSA has made a slight oversight in this regard.

“The N.S.A. came up with more dead ends than viable leads and put a premium on collecting information rather than making sense of it. The N.S.A. fed its bounty into software that created a dizzying social-network diagram of interconnected points and lines. The agency’s software geeks called it ‘the BAG,’ which stood for ‘big ass graph.’”

Either way, Harris presented some serious allegations against the N.S.A. Thanks to Orwell, the country experiences a collective shudder anytime personal privacy is compromised for the greater good of the state.

Or does it? One of Harris’s biggest concerns is that the American people are failing to be outraged at this apparent blatant privacy invasion. Public outcry shut down the comparatively liberal T.I.A. in 2003. Yet today, nothing is being done to rein in the N.S.A. We freely share significant amounts of information on social media sites like Facebook and Twitter. Perhaps ‘sharing’ has become so commonplace that society no longer minds if they involuntarily share information with the government.

Following 9/11, there was a concern that the government would take preventative measures too far. Those thoughts and emotions have dissipated to a large extent in the last eleven years and the country’s activists have moved on to more pressing socioeconomic issues. Harris warns that if we are not careful, the government will inflict a state of constant observation upon its people, a state which thanks to big data storage is now possible.

Related Stories

How DARPA Does Big Data

Seven Big Winners in the U.S. Big Data Drive

World’s Top Data-Intensive Systems Unveiled