Skip to content


Software For Everyone

We talk a lot about hardware here at BookLiberator, it is what we spend most of our time on after all, but it is time to shine a light on the software behind the scenes that turns our page images into beautifully produced “book” collections. That software comes in two parts, scantailor, written by Joseph Artsimovich and djvubind, written by strider1551 of DIYBookScanner.

Scantailor takes the page images from your camera’s memory card:

Page from Concerning Beards

and turns them into nicely cropped, rotated, and white balanced images like this:

Processed image from Concerning Beards

Djuvubind takes all of those individual images, stitches them together, and compresses that into a very tiny book in the djvu format. I have 1400 page academic books that are now pleasantly readable 10 MB files thanks to this combination of Scantailor and Djvubind.

All of this happens automatically. For each of those 1400 page books all I had to do was 1) rotate the first two pages, 2) hit “Go” for auto crop, 3) draw a box around the few pictures so that their full resolution would be preserved in the final output, 4) run djvubind.

Very simple, very easy. When djvubind, which is less than two weeks old, gets the last kinks out, it will be possible to use the same 4 steps to get a tiny book full of beautiful page images which also has a layer of OCR embedded for text searching.

For anyone who has been waiting to get into personal book scanning until the software develops, wait no more.

Crossposted with BookLiberator

Responding to the gatekeeper theory of author’s rights

This post began as a reply to John Degen’s blog post about the Book Liberator. In particular I want to respond to the idea that photographing books is somehow an attempt to steal control of a book’s soul from it’s author, that doing so is a violation of human rights as set forth in the UN Declaration of Human Rights, and that every act of copying is a clear violation of copyright. Readers interested in other responses to the piece can find them in the comment section of John’s post.

In Re Soul Stealing

We seem to be talking at cross purposes here and I think some of that stems from different beliefs about control.

In your original post you talk about the difference between a “work” and the “copies” of that work. The overall thesis seems to be that the author retains control over the idealized “work” as a basic human right and that they make money as authors by selling little bits of that control as permission, permission to print a hardbound edition to one person, permission for a movie adaptation
to another, etc.

There are a couple of complications with this “author as gatekeeper” picture of copyright. There are other rights to consider, like the right we all have to share in the cultural life of the community, a right mentioned in the very first clause of that UN declaration article to which you refer. Even focusing only on the rights of the author, it is not as simple as saying that authors’ interests should be acknowledged and concluding that the particular set of copyright policies we currently have are either the most appropriate or effective means of securing those interests. The EU vs. US dispute about Moral rights is one longstanding example of such a disagreement, and Cory Doctorow is not the only example that giving away things in a digital world is a more effective way to make money than attempting to control each use.

The free software movement, and the creative commons licenses offer a different view of what control might mean to an author. I find it particularly interesting that the most basic requirement of all creative commons licenses, the requirement to acknowledge the author in all uses of a work, is also one of the Moral rights enshrined in European legal tradition. If you write a play, there seems to be wide public agreement that you should be credited as the author of that play. Whether you get to charge someone money to perform your play, or control how much they charge for tickets, who gets to see it, etc, those are not things we all agree about, and they never have been.

We have always had a more nuanced set of rules for how copyright works than the simple gatekeeper model describes. The VCR, Tivo, tape deck, and CD ripper are all examples of common consumer devices whose entire purpose is making copies without asking for particular license to do so. Scale and purpose do matter. That is why we have legislation like the Audio Home Recording Act and court cases explaining that recording TV shows to watch them later is an acceptable use of new technology. Under a simple gatekeeper theory, all of these uses are tantamount to theft and each one is a violation of a recognized human right. Judging by the widespread adoption of these technologies over the last 40 years, that is not a position for which we can assume universal support.

As to the BookLiberator in particular, we can discuss whether people digitizing their own books for purely personal use is a problem under our existing copyright. I offered a number of points for that discussion: most works that I care about are not available digitally, digital versions of my faded works are easier to read, digital versions are also more portable and accessible. I would also like to point out an essay on DRM in the eBook market and how the technological restrictions that publishers place on “licensed” eBooks take away many of the rights we have historically enjoyed with the physical books that we purchase.

The rules for format shifting books are going to be an important topic of discussion over the next few years, whether we come to that discussion because we are talking about digitizing print books or because we need to convert eBooks from our old e-Reader format to something our cell phones can understand. This is a discussion we need to have, but it is not as simple as the gatekeeper model makes it out to be and the human rights you refer to for support are not so one sided as you make them seem, nor are they wedded to the particular copyright statutes that will inform our discussion. As always, copyright is a balancing act.

Update on 2010-08-25: Consumer digitization appears to be coming first to Japan.

Bittorrent and Miro, a better Distributed Proofreading

If you spend some time in the ebook community you inevitably run into Distributed Proofreading, the collaborative proofreading group that supplies Project Gutenberg with high quality text versions of Public Domain books. They are a small community of dedicated editors doing good work. Unfortunately, they are also becoming irrelevant to most of the issues in the field because their multi-layer workflow is simply too slow. When organizations like Google are releasing a million books at once, it is hard to stay relevant when struggling to complete your project’s 20,000 book, even if those books, unlike Google’s, are meticulously verified and formatted. Scale and quality both matter and, if we structure it right, we can rework our communal digitization projects to get both.

Currently, Distributed Proofreaders only releases books after spending weeks or months verifying that the text version matches the original page images. The industrial scanning efforts like Google Books and the Million Books Project generally skip verification entirely and distribute raw text versions with the photographic page images. This is perhaps the greatest key to their large size. Yes, they also paid for large scale scanning but scanning is easy compared to proofreading, and getting getting easier all the time. You can be sure that Google’s library would not be half so large if they had to pay for the kind of quality that Distributed Proofreaders provides. Unfortunately, if the price of this quality is only having thousands rather than millions of books, it is too high to continue paying.

I propose a middle road between the raw image release and the meticulous text one. What if we distributed raw image and unverified text files from day one, but build our distribution network to enable everyone downloading a copy to upload corrections and share those corrections automatically with everyone else who has a copy? If we did that we could gain speed and scale while also building our community of contributers.

Technologically, bittorrent and a rich client like miro would get us most of the way there. We would make each book into a miro channel that people would subscribe to when downloading the book. Once downloaded we would need a book reading view that we could optimize for whatever common reader actions relate to proofreading. Things like spell check and revealing the text around a section to verify academic citations spring immediately to mind. The key is that corrections should come primarily from people’s normal interactions with the books they are interested in, no altruism or active volunteering necessary. Once people have corrected their local copies, the client sends those corrections back to the central server where they can be sent out via rss to everyone subscribed to that book’s channel.

As far as the user is concerned, she simply downloads the books she is interested in with her miro-based library manager and either fixes errors as they bother her, or leaves them alone and watches the text gradually correct itself as other people interested in the same books notice and correct errors. If the errors are really frustrating, she can always fall back to reading the page images and be no worse off than if reading on Google Books or any other large page image-based digital library.

As far as the community is concerned, we get a larger pool of potential contributers because now everyone with a copy can contribute back, and people are able to contribute by sharing spare hard drive space and unused bandwidth rather than having to donate funds to pay for central hosting and distribution. There are plenty of people in the community who have no time or inclination to proofread but would gladly download some book images and leave a torrent running in the background to help share the files more widely.

Making it easier to contribute increases the effectiveness of the project as a whole by helping make sure that all the people who care about a book have the opportunity to put their time into preserving that book. The more people care, the more work gets done. In two years of talking with people about my own book digitization projects, I have grown to have a healthy respect for how much people care about their own books and about preserving them, in whatever form.

In the end, there are only two scalable digitization strategies: teach computers to read, or harness the passion people have for their books for the benefit of us all. A handful of highly organized editors like the Distributed Proofreaders community will always have it’s place, but they cannot handle the scale of this project alone. We should make sure they have some help.

(Crossposted with bookliberator)

Re-making friends

Shortly before christmas last year I had an experience with the internet that left me speechless. The actual effects, besides possibly this blog post, aren’t exactly negative, but the implications of it, and the incredible ease with which it happened, left me a little stunned.

To put it simply, my past caught up to me. A friend, who I had trouble even recognizing at first, friended me on facebook. This one connection was enough to fill my notification streams with the faces of people I hadn’t seen since childhood and pictures I was in before I knew how to read, let alone had heard of something called the “Information superhighway”. My newly rediscovered friend was one of a small handful that I had actually gone to school with continually from Elementary school through High School. Growing up in the NYC School system, that was natural. Equally natural to me was losing touch with most of the other kids as we moved up and between different schools. But now, by reconnecting to a single relationship that ran all the way back in school, I was around them again. Curious, I dove into the stream of new profiles.

What I found was part photo album and part class reunion. Old friends were getting back together, talking about old times and updating each other about all the new things happening in everyone’s lives. Old pictures were scanned and reminisced about, any relationship status of “married” received obligatory public comments of impressed congratulation, and a couple people threw up class photos from back in elementary school, tagging most of the names with either the person’s direct facebook link, or just their name if no one was yet connected to them. That is about when things started getting weird for me.

Something about seeing everyone’s picture streams mesh together Jr. High school, College, Elementary school, and wedding pictures side by side made me a little uneasy. Coincidentally, I had seen one of the posted class pictures a few weeks before while at my mother’s house for the holidays. Looking at the electronic version I realized that it had more people identified than I had been able to identify when trying on my own. On the first grade class picture I couldn’t even find my face without the tag someone put on it. In some ways, facebook knew these events from my life better than I did.

That’s when I realized that this was not just a class reunion or looking through an old photo alum, this was history rebuilding. The activity is much the same, you get together with your friends, talk about the old days and who we all used to be, but when it happens in a social networking site the result is very different. This time, when you reconstruct what happened way back when, it stays reconstructed. The more people that join in, adding details, fleshing out stories, agreeing and disagreeing about how things happened, the better the history becomes, until our childhood photos sit side by side with our wedding portraits in the public profile of our lives and arguments we have forgotten are summarized and immortalized on someone’s Wall.

I was stunned at the ease with which social networking tools allowed this kind of collaborative memory rebuilding to happen. If you had asked me two months ago how to go about getting in contact with my 3rd grade teacher, I would have stared at you blankly. Now, I would just go and reply to her facebook mail (I will soon Ms. Santiago! I just don’t like facebook mail). I almost dismissed the original friend request as SPAM, but the strength of our mutual connection to one of my/our Jr. High teachers made me stare at my friend’s picture until I could put an identity to the face.

If you are a big user of social networking tools you are unlikely to be shocked by anything I’ve said. Even as an arms-reach user I understand that this is simply the tool working as advertised, and I’ve always known that this kind of full-life documentation was possible. I’ve known that it was possible but I always thought of it as a problem for the next generation, the one that is growing up inside the social network right now as the only natives in a strange land. What I didn’t see, or didn’t let myself realize, is just how fast the network is filling in our lives behind us. To a large degree, the online reconstruction of our pasts is happening whether we participate or not.

My childhood happened before the internet, most of my life occurred before digital cameras became mainstream, and I managed to get out of college without anything that current social networking users would recognize as a profile. But the network is there with family and friends on it and the old connections can be re-connected as easily as new ones can be built. If I am going to have to live with this in my lifetime, without the experience or established social norms that would have come along with growing up with such a system, I want good tools and a deeper look at what kinds of things our society might have to change in order to keep up.

The whole experience greatly increased my interest in ideas for a free, and truly user-controlled, social network as well as in the book on privacy, the law, and networked society that I’m attempting to work with Eben on over the course of this semester. More on both later.