Initially, I was reluctant to participate as well, but some of the census advertising, and a little independent research, convinced me it was a good idea. Ironically, the advertising convinced me to participate not by explaining how necessary the census is but by highlighting it’s uselessness.

The ads that struck me are from the subway and follow this pattern: How will we know how many ______ to provide is we don’t know how many people there are? Where the blank can be anything from “hospital beds” to “teachers” to “trains”. It is a sensible plea highlighting the relationship between having reliable information about the beneficiaries of government services and the effective administration of those services. Unfortunately it is also obviously outdated.

Do we actually rely on the census figures, taken once every ten years, to plan out how many trains to run or how many hospital beds we need? I certainly hope not. Operating a transit system or hospital in the 21st century involves collecting records more detailed than the census as a daily part of functioning. You simply cannot manage a train schedule or service changes without accurate knowledge of how many people use what trains at what times, nor can you manage hospital scheduling and inventory without knowing how many people needed what medical resources on each day of your management cycle.

The administration of government services does not depend on the information collected by the census, it produces far more accurate and detailed records than the census is set up to collect. If you were worried about the government having information about your private life, don’t worry about the census. Take some of that energy and consider what the government learns about you every time you use a metrocard or pass a toll booth with your ez-pass, or when all our medical records are digitized and centralized. If you believe that not filling out the census will blind the government to the private details of your life, you need to take a better look at the details they already have.

The census is not about spying on you, it is about enfranchising you. The only government service that is apportioned by the census is representation in the national government, and it is the one that determines how much weight all of your concerns and needs for other services have for the next ten years. So I was glad to be counted and encourage anyone else who has avoided the census thus far to stand and be counted as well.

Hopefully, next time around we can dispense with the ritual paperwork and use the information we already have to, more accurately, estimate population, automatically adding millions of the poorest and most vulnerable members of our community to the count. Like most efforts to enfranchise the poor and vulnerable, it is going to be an uphill struggle.

Currently, Distributed Proofreaders only releases books after spending weeks or months verifying that the text version matches the original page images. The industrial scanning efforts like Google Books and the Million Books Project generally skip verification entirely and distribute raw text versions with the photographic page images. This is perhaps the greatest key to their large size. Yes, they also paid for large scale scanning but scanning is easy compared to proofreading, and getting getting easier all the time. You can be sure that Google’s library would not be half so large if they had to pay for the kind of quality that Distributed Proofreaders provides. Unfortunately, if the price of this quality is only having thousands rather than millions of books, it is too high to continue paying.

I propose a middle road between the raw image release and the meticulous text one. What if we distributed raw image and unverified text files from day one, but build our distribution network to enable everyone downloading a copy to upload corrections and share those corrections automatically with everyone else who has a copy? If we did that we could gain speed and scale while also building our community of contributers.

Technologically, bittorrent and a rich client like miro would get us most of the way there. We would make each book into a miro channel that people would subscribe to when downloading the book. Once downloaded we would need a book reading view that we could optimize for whatever common reader actions relate to proofreading. Things like spell check and revealing the text around a section to verify academic citations spring immediately to mind. The key is that corrections should come primarily from people’s normal interactions with the books they are interested in, no altruism or active volunteering necessary. Once people have corrected their local copies, the client sends those corrections back to the central server where they can be sent out via rss to everyone subscribed to that book’s channel.

As far as the user is concerned, she simply downloads the books she is interested in with her miro-based library manager and either fixes errors as they bother her, or leaves them alone and watches the text gradually correct itself as other people interested in the same books notice and correct errors. If the errors are really frustrating, she can always fall back to reading the page images and be no worse off than if reading on Google Books or any other large page image-based digital library.

As far as the community is concerned, we get a larger pool of potential contributers because now everyone with a copy can contribute back, and people are able to contribute by sharing spare hard drive space and unused bandwidth rather than having to donate funds to pay for central hosting and distribution. There are plenty of people in the community who have no time or inclination to proofread but would gladly download some book images and leave a torrent running in the background to help share the files more widely.

Making it easier to contribute increases the effectiveness of the project as a whole by helping make sure that all the people who care about a book have the opportunity to put their time into preserving that book. The more people care, the more work gets done. In two years of talking with people about my own book digitization projects, I have grown to have a healthy respect for how much people care about their own books and about preserving them, in whatever form.

In the end, there are only two scalable digitization strategies: teach computers to read, or harness the passion people have for their books for the benefit of us all. A handful of highly organized editors like the Distributed Proofreaders community will always have it’s place, but they cannot handle the scale of this project alone. We should make sure they have some help.

(Crossposted with bookliberator)

Here are the two of them side by side. The guru plug is the smaller black one.

The Guruplug is an upgrade from the original Sheevaplug development kit in pretty much all respects. It has a more powerful processor, a much expanded array of ports, making it more capable, and is smaller in all dimensions.

The only thing that new Guruplug owners might miss from the older model is the full size SD card slot on the side. Given that this has been replaced by: 1) a microsd slot, 2) an additional USB port, which could easily take a SD card inside of a SB adapter, and 3) an eSATA (Gb/s) cable for connecting external hard drives at their native speeds, I don’t think many people will actually miss the old SD slot.

Now Martin just needs to get his so he can teach the Debian installer about the new hardware, and we’ll be able to put it through some paces.

I’ll be spending the meantime reading up on how to shape local network traffic so I can replace my router with my new Guruplug.

Connections in Context

Social networks, whether online or off, are made up of person-to-person connections. We each belong to many of these networks, and every one has it’s own shared history and set of expectations about what information is appropriate to communicate and with whom. As social animals we navigate these social contexts in all of our offline communications. We select which facts about a romantic date we should share with family members, or drinking buddies, or work colleagues, with a practiced familiarity born from years of inter-personal interaction.

That same easy information management is what we need for our digital communication tools. Existing tools do this poorly, if at all, because they tack information management on after the fact. We need to realize that social contexts are what make a "social network" different from any other computer network and start building an awareness of context into our communication tools from the ground up.

These then are your minimum requirements for a social network: 1) the ability to make person-to-person connections and 2) the ability to manage what information you communicate based on the social context of the personal connection.

Building for Flexibility

Once people can connect with each other and manage what information they communicate, they’re going to need some actual communication tools. Social networks are not communication tools, they enable communication in much the same way that roads enable travel by car but don’t actually do the driving for you.

Online, communication tools can take many forms. IM, Email, Wave, bittorrent, photo sharing, wikis, blogs, rss, these are all possible ways that people will want to communicate with their friends and family. Ideally our social networking tools will be flexible enough to allow any of these kinds of communications, or whatever new ones we come up with down the road.

If I could get one message out to the various groups making distributed social networking systems it would be this: remember that the communication tools you are building around will change. If your communication tools and your social network are too closely intertwined, you run the risk of losing both when people decide to switch to a cooler communications tool or when they want to join their friend’s social network rather than get their friends to switch to your tools.

One way to get network flexibility, as I mentioned in my last post, is to separate your social networking layer and your communication tools entirely. Make social network connections more like an ssh connection or a VPN with built in information management and let people run whatever they want over the network. Maybe instead you want to build a central communication tool with a flexible plug-in architecture. Perhaps you want to build a protocol for different communication tools to talk to each other and let the everyone else build to your specifications.

It is easy to think, when building a whole system for social networking, that your tools are the only tools people will use. It is important to remember that there are lots of people building these systems, using lots of architectures and writing in lots of languages. The more flexible your system, the better able it will be to benefit from the work going into these other tools by incorporating or cooperating with those tools rather than competing with them for a fragmented user base.

Network Effects

Fragmented user bases are a particular problem for social networks because social networks only work when you and your friends are in the same one. Currently ~0% of users are on a distributed social network. 100% should be, but getting from here to that Kantian paradise will be difficult. If you are going to try, you need a plan.

I met with the Disapora developers last week, before they had quite hit their summer fund raising goal, and their plan sounds very sensible. The idea is that your distributed social network system starts our for most people as a client for their existing social communication tools, whether those are social networks like Facebook or photo sharing sites like flickr. You manage everything as if your friends were on Diaspora with you but you can also continue communicating with your contacts who are using the old tools, sending status updates through Facebook and posting new pictures through flickr. You never have to choose whether to abandon your existing social connections in order to change your communication tools.

Once some of your friends do start using the new system, things change. When your two machines notice that they are both able to run a distributed social network, they stop sending your communications through the old centralized services and start communicating over direct encrypted connections. As more of your friends switch over, more of your communications become private.

It is a nice gradualist strategy that neatly inverts the network effect problem. We could all start running our own network and still talk to people on Facebook freely, but every person who moves to our network hides more communication from Facebook by joining. Facebook has no place in our network of trusted friends because they are neither trustworthy nor our friends. When we have tools for direct private communication, Facebook drop out of the picture.

However you feel about this particular plan, as I have loosely paraphrased it, every distributed network builder should have something like it at the core of their system design. We’ve had the capability for private darknets for a long time now and people have not flocked to them. We need tools that people will actually move to if this is going to work, and that requires a plan.

Help Your Friends

Facebook grew at the expense of it’s users privacy. If we build our system wisely, we can build a tool that not just salvages, but actively promotes the privacy of its users. We should do this for our own sake, but we should also do this for our friends. That’s the difference between our tools and theirs: we’re building for people we care about, not people we need to exploit in order to make a profit. That difference should be clear in every part of our system.

Posts in this series

Part 1 – The Idea: Freedom Box

Part 2 – Finding each other: Dynamic DNS Facebook

Part 3 – Talking amongst ourselves: Friend-to-Friend Network

Part 4 – Putting the pieces together: Freedom Box schematic

Part 5 – Making it easy: Look and Feel forthcoming

Build less.

What we need is a tool for making connections with other people and a social firewall to manage what parts of the information you put in the system get sent to which people you know.  These are the pieces that hold a “social network” together and make it different from any other digital communication tool.  The rest, all the specific communication tools like email, IM, photo sharing, VoIP, etc, are being bundled together in a dozen permutations as we speak. People will continue building such systems indefinitely.  That is about all about using the network, not building it. Once you’ve assembled a social network it should no more care how we communicate over it than the internet does.

Most new social networks fail because the costs of switching and having to rebuild your social connections through the new tool outweigh whatever advantages that tool might have to offer. When we separate the communication tools from the underlying social network, we will gain the flexibility to experiment that we need without sacrificing our personal relationships to the alter of bleeding edge communication software.  So build that part first. If you get it right, you will become a part of everyone’s social software stack, not just the one in the Freedom Box.