Scantailor takes the page images from your camera’s memory card:

and turns them into nicely cropped, rotated, and white balanced images like this:

Djuvubind takes all of those individual images, stitches them together, and compresses that into a very tiny book in the djvu format. I have 1400 page academic books that are now pleasantly readable 10 MB files thanks to this combination of Scantailor and Djvubind.

All of this happens automatically. For each of those 1400 page books all I had to do was 1) rotate the first two pages, 2) hit “Go” for auto crop, 3) draw a box around the few pictures so that their full resolution would be preserved in the final output, 4) run djvubind.

Very simple, very easy. When djvubind, which is less than two weeks old, gets the last kinks out, it will be possible to use the same 4 steps to get a tiny book full of beautiful page images which also has a layer of OCR embedded for text searching.

For anyone who has been waiting to get into personal book scanning until the software develops, wait no more.

Crossposted with BookLiberator

Currently, Distributed Proofreaders only releases books after spending weeks or months verifying that the text version matches the original page images. The industrial scanning efforts like Google Books and the Million Books Project generally skip verification entirely and distribute raw text versions with the photographic page images. This is perhaps the greatest key to their large size. Yes, they also paid for large scale scanning but scanning is easy compared to proofreading, and getting getting easier all the time. You can be sure that Google’s library would not be half so large if they had to pay for the kind of quality that Distributed Proofreaders provides. Unfortunately, if the price of this quality is only having thousands rather than millions of books, it is too high to continue paying.

I propose a middle road between the raw image release and the meticulous text one. What if we distributed raw image and unverified text files from day one, but build our distribution network to enable everyone downloading a copy to upload corrections and share those corrections automatically with everyone else who has a copy? If we did that we could gain speed and scale while also building our community of contributers.

Technologically, bittorrent and a rich client like miro would get us most of the way there. We would make each book into a miro channel that people would subscribe to when downloading the book. Once downloaded we would need a book reading view that we could optimize for whatever common reader actions relate to proofreading. Things like spell check and revealing the text around a section to verify academic citations spring immediately to mind. The key is that corrections should come primarily from people’s normal interactions with the books they are interested in, no altruism or active volunteering necessary. Once people have corrected their local copies, the client sends those corrections back to the central server where they can be sent out via rss to everyone subscribed to that book’s channel.

As far as the user is concerned, she simply downloads the books she is interested in with her miro-based library manager and either fixes errors as they bother her, or leaves them alone and watches the text gradually correct itself as other people interested in the same books notice and correct errors. If the errors are really frustrating, she can always fall back to reading the page images and be no worse off than if reading on Google Books or any other large page image-based digital library.

As far as the community is concerned, we get a larger pool of potential contributers because now everyone with a copy can contribute back, and people are able to contribute by sharing spare hard drive space and unused bandwidth rather than having to donate funds to pay for central hosting and distribution. There are plenty of people in the community who have no time or inclination to proofread but would gladly download some book images and leave a torrent running in the background to help share the files more widely.

Making it easier to contribute increases the effectiveness of the project as a whole by helping make sure that all the people who care about a book have the opportunity to put their time into preserving that book. The more people care, the more work gets done. In two years of talking with people about my own book digitization projects, I have grown to have a healthy respect for how much people care about their own books and about preserving them, in whatever form.

In the end, there are only two scalable digitization strategies: teach computers to read, or harness the passion people have for their books for the benefit of us all. A handful of highly organized editors like the Distributed Proofreaders community will always have it’s place, but they cannot handle the scale of this project alone. We should make sure they have some help.

(Crossposted with bookliberator)

The longer answer is that it is a device I’ve been working on since the summer to more easily convert my paper books into a digital form. I’ve had test hardware working for a number of months but things were going pretty slowly until James decided to build some image processing scripts to accompany the effort. Those scripts became a fully fledged python application around the end of the year, and we’ve since begun documenting the project in earnest.

The bkrpr wiki has all the relevant links, and a nice front page YouTube video of the device in use. Or a very poor YouTube video of the inside of my room, depending on how you look at it.

If you’re curious to check it out, take a look at the site, or grab the processed test pages online or in pdf.

Looking back

Before we mourn losing the word processor we should look back at what it is and what it has done for us. At its core the word processor is virtual paper and the promise of WYSIWYG, the promise that, however you arrange things on the virtual paper, they will look the same on real paper when you print a copy. Other capabilities were built in later, things like change tracking, macro languages, and outlining modes, but these tools never take the spotlight; word processors remain word processors, not outlining tools or version control systems. It is all about the virtual paper.

And it was great. In Jr. High we spent weeks learning how to properly format documents. There were tests on where to place the opening line on a business letter, how many lines to skip between address blocks and the To: or From: lines, and other layout details. We were effectively learning typewriter office skills. Word processors made laying out documents so easy that simple formatting information like this could be stored for us. So, by the time any of us had to write a business letter, we no longer needed to remember how to format it. All we needed to do was pick “business letter” from the template menu of our word processor and remember to replace all the dummy "YOUR NAME HERE" text with our real information.

Problems with virtual paper

It was great, but there were problems. Competing word processors were often incapable of reading each other’s files, locking people into one camp or another. Time and competition between camps brought new versions of the word processor software; new tools were added to allow for more complicated layouts, to help correct common errors, and to make existing features easier to use. These changes increased what people could do but also brought further incompatibilities. New versions of the software had problems using old documents and documents in the new formats wouldn’t work at all with the old software.

Virtual paper began to age. People who had used these virtual sheets as a way to archive documents found that new word processors would corrupt the formatting in old documents and refuse entirely to open some of them. People who wanted to switch from one word processor camp to another had it worst, often having to rely on third party conversion utilities to use their old documents at all.

Digital communication

So virtual paper aged, people got increasingly tied to one format or another, and the internet happened. Now rather than exchanging the paper documents, people began exchanging the virtual paper versions, making it even harder to know what version of what program your document would be opened with. Once .doc and .wpd, and later, .odt documents were being sent around, the promise of WYSIWYG stopped meaning much. What you saw might be what you got but you could no longer know just what the other person was getting.

To regain control of their formatting, people began using PDF, the virtual printer for our virtual documents. By making a PDF of your document you basically trade the ability to edit your document in the future for an assurance that your document will look and print the same on any other computer. While this is useful for documents where formatting is important, resumes are an often cited example, it represents a step backwards for word processors.

Word processors are not the only, or even best, way to generate PDFs. Once PDFs became a standard form of print-ready documents, people built web applications to generate them from any page on a website as well as OS-level PDF printers that let you create a PDF out of any file on your computer. As the tools capable of producing print-ready documents multiplied, the word processor began to lose its place as people’s primary tool for authoring documents.

Simplification

At the same time that PDF was ironing out incompatibilities and creating a nearly universal form of print-ready virtual paper, people started to notice that a lot of their print-ready documents were never getting printed. As people grew more comfortable with digital means of communication they began to rely on them to carry more of the content once invested in paper. People who had once felt the need to attach word processor or PDF documents to their emails began moving the material from those documents into the email itself. Personal correspondence moved not only to email but to chat rooms and instant messaging, places where printing to paper was not a concern.

Free of the complicated formatting necessary for paper, people fell back to the handful of basic formatting options they felt most necessary for communication in a digital context, things like:

>> quoting and
> later
/italics/
*bold*
[ links | to places ] and

paragraph breaks.

Those five, in addition to the rich complexity of our natural languages, turn out to cover most of what people need for communicating the sense of their messages.

This trend, of replacing elaborate formatting, like virtual paper, with lightly marked up plain text, is shrinking the domain of word processors each day. Wikis, the largest document creation projects in history, all use variations of the basic formatting options shown above. Current social networking and publishing tools are the same, as is email. The rise of syndication formats like RSS and ATOM is perhaps the best example of people happily removing all page layout formatting to more easily access the plaintext or lightly marked up text underneath.

Which is not to say that people stopped caring about the visual appearance of their documents. Layout and design remain as important as ever, but all of that information simply moved to the side, into CSS sheets, and blog or social networking theme packages.

The prime example of both these trends is the web itself. While many PDF documents, and some few .doc files, remain available online, they are dwarfed by the HTML ones that make up the web around them. The combination of HTML and CSS has beaten out all previous electronic formatting standards and become the most universal way to format writing since paper. Word processors have never created either format well.

Today

Today I neither send nor receive business letters. Almost all of my professional communication happens over email, including things like negotiating for event space, arranging travel, and handling requests for legal services from our office. I write my documents in markdown, one of the popular lightweight markup systems, and use a program called pandoc to convert them to whatever format I need: PDF, HTML, LaTeX, or odt. Separate style sheets let me easily format the same text for letterhead, publishing on our website, publishing on my website, or general printing. The things I read come to me as websites, RSS feeds, email messages, and, most recently, ebooks (which are really just simple HTML documents re-packaged).

And so the word processor has passed from my life. I realize that it has not passed from everyone’s and it may not have left yours yet, but if the trends towards simple markup and PDFs continue forward, odds are that it will.