NEW YORK // In a hastily convened meeting, federal law-enforcement and intelligence officials briefed congressmen who demanded more details about the secret data-collection programmes that were disclosed last week.
As he emerged from the meeting on Tuesday, Brad Sherman, a California Democrat who sits on the House of Representatives' intelligence committee, appeared stunned.
While legislators recently renewed the laws that allow the National Security Agency (NSA) to collect internet and phone records, Mr Sherman said he and other members of Congress "did not know one billion records a day were coming under the control of the federal executive branch".
Apparently, even those who are supposed to have oversight of America's spy agencies had been unaware of aspects of the vast expansion of the NSA's capabilities over the past decade. Catalysed through rapid advances in data processing, the agency's ability to monitor people across the globe have been transformed by access to vast amounts of the 2.5 quintillion bytes of digital data created daily.
Combining the rapid advances in "big data" analysis software with the capacities of ever-expanding super computers and storage facilities, the NSA is able to churn through the massive amounts of online data with algorithms that identify patterns of association and even forecast behaviour.
The leak of what appeared to be routine secret requests for domestic telephone records made on one major carrier, as well as the Prism programme of obtaining information directly from major web companies, shed light on just how embedded data sweeping has become - in the US and abroad.
Because the requests are secret, as are the special courts that rule on them, it is not known exactly what the government does with this data. Most likely, though, much of it is "social-network analysis: trying to find out the networks of communication and structure of organisations by the way people talk to each other", said Ben Hammersly, a London-based fellow with the Brookings Institution think tank's Center for 21st Century Security and Intelligence.
Without having to eavesdrop on telephone calls, NSA analysts, through their databases and software, are able to piece together a picture of an individual's life by running the various streams of his or her online presence - such as phone call locations, credit card purchases and Facebook updates.
"The new thinking is that people are the sum of their social relationships, online interactions and connections with content. To fully investigate an individual, analysts need to look at the widest possible penumbra of data that surrounds the person - not just whom they know, but whom those people know too, and so on," wrote the authors of a book on the subject, Big Data.
The NSA, with classified budget reported to be US$10 billion (Dh36.7bn) last year, has been growing apace. It will soon open a new $2bn facility in the Utah desert that will store nearly immeasurable amounts of people's online data, according to James Bamford, a reporter who has written about the NSA for decades.
The NSA is also racing to build what will perhaps be the world's most powerful supercomputer to break through encryption codes on data pulled from the so-called "deep web" of government documents and other secret information, Bamford reported last year in Wired magazine.
"The more data they have and the better the computational capabilities the more sophisticated the algorithms can be, and the better they can do," said Irving Lachow, the director of the Technology and National Security Program at the Center for a New American Security, a Washington-based think tank. "So over time they may get very, very good at this."
Big data will play an increasingly central role in national security work just as it is in the world of private industry, Mr Hammersly said.
But the revelations about the NSA's abilities have outraged civil liberties groups and some legislators because of what appears to be massive privacy infringement that goes beyond what the law allows.
Part of the problem is that "we're running into a situation where the technology is outpacing the legal frameworks we've put in place to provide these protections", said Mr Lachow.
An example of this mismatch is the issue of the telephone record metadata that was requested by the government. Since it did not request the content of people's calls, only the numbers and timings, a warrant as part of a specific investigation is not needed.
"Because there is so much data out there now and given the analytic capabilities it is now possible to make all kinds of conclusions [with metadata] about people's behaviour and identity that wasn't possible five or 10 years ago," Mr Lachow said. "You're constantly balancing the probability of a false alarm with the probability you missed something."
While Barack Obama has said he encourages a renewed debate on the legal issues around privacy, the secrecy around the programmes also makes having an honest debate in the public sphere nearly impossible, Mr Hammersly said. "We have to decide what techniques are compatible with a free society" but at the moment it is "impossible to have a discussion about the ethics of the situation", he said. "I imagine East Germany was an incredibly safe place to live but I don't think that's the sort of society under which we wish to live."
tkhan@thenational.ae