This week we have looked at the three main elements of the NSA’s surveillance system: Bulk data collection and the construction of an index for all communications in the country, use of private companies to store and process the content of our domestic data, and partnerships with other government agencies at home and abroad. We have examined all of these elements to so that we can try and judge the NSA’s surveillance system based on how it is constructed rather than by the motives and ideals of those currently using it. Now that we have examined the components, it is time to look at the bigger picture.

Technology of Power

Wholesale collection of data, use of private companies as data refineries, and partnerships of mutual convenience with other government surveillance agencies. Those are the functional components of the NSA system, the bits of code out of which it is built. What does that tell us about the system as a whole? We know that tapping into fiber optic lines naturally leads to wholesale data collection. We know that during wholesale data collection it is difficult or impossible to tell just whose data is being collected. We know that possessing all of the data turns what were once external checks and balances, like the prohibition on the NSA collecting US citizen data, into matters of self-policing and internal procedure design. We also know that, given all of this for a decade, the NSA has sought to increase how much data on US citizens they can search, radically increase how long they can keep data, and expand partnerships with groups that can volunteer information for the system that is free of any regulations. Now that we know that, we can ask the real question: is this going to be the kind of system we use to police democratic societies for the rest of our lives?

Before you decide, take a minute and watch this talk. The speaker, Malte Spitz, is a member of the German Parliament and used the German freedom of information laws to get a copy of all the “metadata” that his phone company stored about him. You can watch six months of his life reconstructed on that video. Everywhere he went, everyone he talked to, and all the groups he spoke with are captured in that metadata. There is power in being able to reconstruct someone’s life like that. Being able to reconstruct everyone’s lives at once is not just powerful, it is the kind of technology that could keep a government in power. Whether the NSA system was built to chase down terrorists or to disrupt political dissent does not matter. The power of the system matters and how much power we are comfortable giving to the secret operators of such a system matters.

In our names the US government is building a new kind of surveillance system, one that upends all the laws meant to regulate such activity and that is tied directly into the internet connections that will be the primary communication infrastructure for the rest of our lives. We have perhaps the best opportunity we will ever get to examine the actions taken in our names and set new rules for how a democratic society governs itself in this area. Our deliberations and decisions will have wide ranging repercussions. As the price of technology continues to fall there will be many others capable of building similar systems and the choices we make now will set the standard of behavior when that happens.

If we push back and we decide that this kind of monitoring is incompatible with a democratic society, our position as the central hub of the global internet means that we can hold that line for the next generation. If we move in the other direction and commit the center of the network to constant monitoring and recording, what will we say when those same tools are used to prop up the next “Axis of Evil” or suppress the next Arab Spring?

In the technology community “code is law” is said as a reminder that our technologies are governed not by our intentions but by the way they are put together. It is also sometimes spoken in a hopeful note because, while code may be law, we write the code. We determine how our technology is built. It can be hard and it can be complicated, but we need to do it because, if we don’t do it right, someone else will do it wrong.

So far this week we have looked at two of the three main components of the NSA’s surveillance system: how the NSA collects raw data from fiber optic cables and uses that to build an index of “metadata” that maps nearly all communications in the country going back to 2001 and how they enlist private companies as data distilleries holding and processing the contents of our domestic data. Today we will finish looking at the functional elements of the NSA system with a look at how government agencies at home and abroad partner with the NSA, skirting all effective data protection regulations as a result.

Sharing is caring

The NSA is a single government agency. It may be the "largest, most covert, and potentially most intrusive intelligence agency ever" and it may sit at the center of the global communications network, but it is still just one agency and it has limits. They are still somewhat prohibited from directly targeting US citizens, which is the only factor limiting which domestic fiber optic cables they can tap into with splitter prisms. They also lack domestic access to the 7.25% of global internet traffic that does not pass through the US during transmission. The essential allies for overcoming these obstacles are other government agencies, both those at home and abroad.

At home the NSA cooperates directly with numerous government agencies, most importantly the CIA, FBI, and the little known National Counter Terrorism Center (NCTC). In addition to sharing expertise, connections, and personnel resources, when these agencies work together they also benefit by skirting around laws designed to control just where they can operate. The NSA’s intelligence gathering is limited by law to foreign communications. In order to collect and store the phone records of purely domestic phone calls, as we can now confirm they are doing, someone other than the NSA must do the collection. In the case of phone records, the FBI is the one actually requesting records from the phone companies. The same is true of PRISM requests for internet communications. In all cases the NSA is the one who stores and analyzes the data; the intermediary agencies are used as legal cover. The reason for this game of digital hot potato is that data that is lawfully obtained by the government becomes fair game for other parts of the government to search. So, once the FBI has obtained everyone’s phone records the NSA no longer feels that the legal prohibitions on collecting data about US citizens apply.

Making it easier for different government agencies to exchange information was one of the main reasons for creating the NCTC in 2003. Initially this information was limited. Information about US citizens who were not suspected of any crime could be included but could not be kept for longer than 180 days. Then in press release last march the Attorney General changed that from 180 days to five full years. Perhaps unsurprisingly this is the same length of time the NSA keeps such data on citizens. This one government partnership alone is a significant expansion of the NSA’s surveillance system. The NCTC brings access to all Federal databases including flight records, financial forms submitted by people seeking federally backed mortgages, the health records of people who sought treatment at Veterans Administration hospitals and many others. The only restriction on what databases the NCTC may keep is that they must be “reasonably believed” to contain “terrorism information.” With databases this large it seems reasonable to believe they contain everything.

When foreign governments cooperate in surveillance even these trivial restrictions fade away. Just as we place no restrictions on what the NSA may do with information about non-US citizens, other governments place no restrictions on what their spy agencies can do with information about US citizens. Theoretically then it would be possible for two nations to spy on each other and then exchange information, much like strangers on a train. By accident or by design, this is much what happens with the British intelligence agency GCHQ, who we help access more than 200 fiber optic cables. In return we gain access to the processed metadata they collect. Any data we wish to share with them can be done through the NCTC. The only difference between our two programs is how long we each keep data. While we keep information on our citizens for up to five years the UK government only stores information on their subjects for a maximum of 30 days.

Tomorrow we will put all these pieces into context and draw some conclusions about what these components mean for the surveillance system as a whole: Part 4 – The End.

Update July 8: We learned over the weekend more details about the GCHQ cable tapping and have now have information about how the Australian and other close international partners operate their own social monitoring stations. The geographical diversity of these partner nations means that nearly all of the undersea fiber optic cables that tie the internet together are open to unregulated monitoring by one of our partners. As other nations build their communication storage capacities to match our own this means it will be legally and architecturally possible for this small group of democratic governments to access complete records of all internet communications. As long as nations only store information about each other’s citizens, no domestic surveillance laws will be triggered. As long as the records are complete, each nation will know that any information about their own citizens they wish to access at a later date can be simply requested from a partner.

Yesterday we looked at how the NSA collects raw data from fiber optic cables and uses that to build an index of “metadata” that maps nearly all communications in the country going back to 2001. Today we take a look at the second component of that system: using private companies to store and process the contents of our data.

Distilling Our Data

By tapping into our nation’s fiber optic cables the NSA has built what is likely the largest data collection tool in the world. It is enough to make the Stasi jealous. Processing through all this data is an immense task and no doubt one reason they are building the world’s largest computer. Until that comes online, the NSA relies on an older method that they call “contact chaining” to search through all the data they collect. Contact chaining is when you start with a single person and look through the NSA index of communications to identify every person they have phoned or emailed. From there you can begin searching each of those newly identified contacts to see who they have phoned or emailed, proceeding out however many degrees of separation you wish until, we can assume, you invariably end up searching through Kevin Bacon’s address book. If this contact chain includes someone the NSA is interested in, one of the FISA judges instructs that person’s email, social network, and other online account providers to turn over all information they have about the individual. This collaboration with our largest technology companies is the PRISM program.

Architecturally, using private companies to store data is a powerful strength of the NSA’s system. Data stored by private companies has almost no legal protection against government search, cost nothing to the NSA to store, and are kept essentially forever. Perhaps most importantly, because all these tech companies make their money by studying our activities for advertisers, the data they produce to the NSA has been tagged, cross-referenced, and refined into useful formats. While this form of “share everything” plan might be objectionable to consumers, and no doubt this accounts for some of the current upset over the NSA’s activities, in the normal course of events the technology companies are not even allowed to disclose whether they have received demands form the FISA court, let alone what data may have been turned over.

Put on a happy face

Access to the data warehouses of Google, Facebook, Microsoft, and others fills a vital role in the NSA surveillance system by turning the organizations we trust with our data into informants against us. While many of these companies may participate in PRISM unwillingly, Yahoo for example sued the government in secret court to avoid participation, part of the PRISM program is no doubt designed to improve relations with these companies and accustom them to providing information. Such positive relationships with private companies can be quite productive for the NSA. In 2001 it was voluntary cooperation from network operators that enabled the NSA to install all those fiber optic splitters, which operated for four months before the panel of judges charged with overseeing NSA surveillance were informed of the program.

Good relationships also encourage some companies to go beyond merely complying with demands for data and actively make it easier to access such data about customers, as Sprint did when building a web portal for police that made it so easy to search for the location of individual phones that it was used 8 million times in 2008 alone. We now know that there are more than 80 companies voluntarily cooperating with the NSA, including one major US network operator that is steering data from around the US past the NSA splitters. It is unclear whether the NSA is gathering credit card information from one of these voluntary relationships or through PRISM demands.

Maintaining positive relationships with the companies participating in PRISM also goes a long way toward preventing those technology giants from making changes that would reduce the amount of information the NSA can access. These technology companies are as close as we currently have to a civil society infrastructure for digital communications. If they were significantly against the NSA’s activities, they could do significant damage to the NSA’s capabilities simply by changing their own business practices. When faced with a similarly board government monitoring program in Sweden, internet providers in the country decided to stop keeping records of user activity so that there would be no information to turn over. Similarly, our own tech companies could decide to keep less information about us, to encrypt more of it by default, or make other architectural changes that would reduce the volume of information they are required to transmit to the NSA. The $100 million dollars the NSA spent collecting data from private companies between 2001 and 2006 likely helps prevent those kinds of changes.

Yet, no matter how cozy the relationship or how extensive a private company’s resources, to build a truly global surveillance system you need the cooperation of governments: Part 3.

If you have heard anything about the NSA this month, you have heard grand statements and sweeping generalizations. More than likely you have heard a whole gallery of commentators try and relate the news to ideals like “liberty”, “security”, and “privacy”, as if we could all agree about what those ideals mean. In the technology world we have a saying, “code is law”, to remind everyone that the systems we build are not governed by our ideals, they are governed by the practical way we put them together. What the NSA has built is a tool: a system of technology, personnel, and regulations. To judge this tool based on the ideals of those involved or the reasons for its creation is a job for pundits. Us? We know to look at the code.

Prisms, internet giants, and James Bond.

So, what exactly is the “code” of a national surveillance system? Unpacking the avalanche of NSA information this month we can see three major components of the system: collection of wholesale raw data, use of private companies as data refineries, and collaboration with other spy agencies, including the British NSA equivalent, the GCHQ. These three components determine how the system works, what its limitations are, and what it is capable of; they are its “code” and they each have important ramifications for the system as a whole so we will look at them each in turn.

Carbon copying the internet

Of all the NSA programs reveled recently, PRISM has gotten perhaps the most press. We will be focusing on the specifics of this program in the next section but it is worth mentioning here for its name alone. Have you ever wondered why they would name a data collection program “Prism”? While the actual reasons are still classified, my guess is that the name is an homage to the NSA’s practice of using actual glass prisms-like devices for data collection.

Glass is useful for data collections because most internet traffic that travels any distance is converted into patterns of light and sent over fiber optic cables. If you can tap into the fiber optic cable you can install a prism-like device * you can split that light, sending part of it further down the line as intended while sending a duplicate copy somewhere else. We learned back in 2006 that the NSA began installing prism-like “splitter” devices in all the major fiber optic cables in the country, installing secret rooms at the nation’s leading phone and internet companies to capture copies of everything flowing over the network.

Notice that this approach is only useful when you want to copy everything going over a cable; you cannot, for instance, have the splitter recognize what information is bound for overseas and what is just moving over to the next town. Once you get down to the actual cables all our communications run through, all our data looks the same. This is fundamentally important because the NSA is legally prohibited from monitoring US citizens but, once you tap into the cables, the only way to make sure that you will end up with the particular data you want is to take all of it and look through it later. While the NSA has varied what portions of this information it keeps, and under what legal authority it claims the right to keep them, those changes are governed by internal decisions at the agency, not by the technology of the system itself.

Your Permanent Record

It is impossible to say just how much of this raw data the NSA has kept since 2001. Because there are no legal restrictions on storing information about non-US citizens, the recently disclosed documents pay little attention to the issue. We have learned that in Germany alone the NSA collects half a billion records a month. One possible indication of the scale of the data being stored is the new $2 billion data center the NSA is opening this September: estimates are that it will be able to store all the traffic that moves over the internet for years to come.

For US citizens we know that the NSA collected a nearly complete index for all emails sent between 2001 and 2011, when they halted the program for “operational and resource reasons”. This index includes a record of each email sent, who sent it, and what computer network they were on when sending it. They appear to have collected some form of credit card transaction history, likely a list of purchase times, amounts, and merchants. Similarly, the NSA has been collecting records of all phone calls made on US carriers, what numbers they call, how long they talk, and, potentially, where they call from if they are using mobile phones. This sort of communications history for an individual has historically been called a “pen register” and government agencies normally need a court order to create one. The NSA argues that they are not governed by these rules because they collect data in bulk and only search through it later while the older laws were designed for devices that did both at once. This recording of phone activity is still going on today.

In the press this index of everyone’s activity is refereed to as “metadata” because it is information about our communications but not the contents of those communications. Storing the contents of our communications would run afoul of wiretapping laws and would require many times more storage than keeping an index does. Until that new data center goes online, such activity might be operationally difficult for the NSA as well as legally treacherous. Instead, the NSA keeps an index of our communications and, whenever they want to see the contents, they request them from the tech companies that run our email and social networks.

Tomorrow we will look at the role that private companies play in distilling our data: Part 2.