Yesterday we looked at how the NSA collects raw data from fiber optic cables and uses that to build an index of “metadata” that maps nearly all communications in the country going back to 2001. Today we take a look at the second component of that system: using private companies to store and process the contents of our data.

Distilling Our Data

By tapping into our nation’s fiber optic cables the NSA has built what is likely the largest data collection tool in the world. It is enough to make the Stasi jealous. Processing through all this data is an immense task and no doubt one reason they are building the world’s largest computer. Until that comes online, the NSA relies on an older method that they call “contact chaining” to search through all the data they collect. Contact chaining is when you start with a single person and look through the NSA index of communications to identify every person they have phoned or emailed. From there you can begin searching each of those newly identified contacts to see who they have phoned or emailed, proceeding out however many degrees of separation you wish until, we can assume, you invariably end up searching through Kevin Bacon’s address book. If this contact chain includes someone the NSA is interested in, one of the FISA judges instructs that person’s email, social network, and other online account providers to turn over all information they have about the individual. This collaboration with our largest technology companies is the PRISM program.

Architecturally, using private companies to store data is a powerful strength of the NSA’s system. Data stored by private companies has almost no legal protection against government search, cost nothing to the NSA to store, and are kept essentially forever. Perhaps most importantly, because all these tech companies make their money by studying our activities for advertisers, the data they produce to the NSA has been tagged, cross-referenced, and refined into useful formats. While this form of “share everything” plan might be objectionable to consumers, and no doubt this accounts for some of the current upset over the NSA’s activities, in the normal course of events the technology companies are not even allowed to disclose whether they have received demands form the FISA court, let alone what data may have been turned over.

Put on a happy face

Access to the data warehouses of Google, Facebook, Microsoft, and others fills a vital role in the NSA surveillance system by turning the organizations we trust with our data into informants against us. While many of these companies may participate in PRISM unwillingly, Yahoo for example sued the government in secret court to avoid participation, part of the PRISM program is no doubt designed to improve relations with these companies and accustom them to providing information. Such positive relationships with private companies can be quite productive for the NSA. In 2001 it was voluntary cooperation from network operators that enabled the NSA to install all those fiber optic splitters, which operated for four months before the panel of judges charged with overseeing NSA surveillance were informed of the program.

Good relationships also encourage some companies to go beyond merely complying with demands for data and actively make it easier to access such data about customers, as Sprint did when building a web portal for police that made it so easy to search for the location of individual phones that it was used 8 million times in 2008 alone. We now know that there are more than 80 companies voluntarily cooperating with the NSA, including one major US network operator that is steering data from around the US past the NSA splitters. It is unclear whether the NSA is gathering credit card information from one of these voluntary relationships or through PRISM demands.

Maintaining positive relationships with the companies participating in PRISM also goes a long way toward preventing those technology giants from making changes that would reduce the amount of information the NSA can access. These technology companies are as close as we currently have to a civil society infrastructure for digital communications. If they were significantly against the NSA’s activities, they could do significant damage to the NSA’s capabilities simply by changing their own business practices. When faced with a similarly board government monitoring program in Sweden, internet providers in the country decided to stop keeping records of user activity so that there would be no information to turn over. Similarly, our own tech companies could decide to keep less information about us, to encrypt more of it by default, or make other architectural changes that would reduce the volume of information they are required to transmit to the NSA. The $100 million dollars the NSA spent collecting data from private companies between 2001 and 2006 likely helps prevent those kinds of changes.

Yet, no matter how cozy the relationship or how extensive a private company’s resources, to build a truly global surveillance system you need the cooperation of governments: Part 3.