Skip to content

technology

A new take on Email to Blog

tl;dr I have a new son, Jasper, and now need to share photos with friends and family. The existing tools (social networks, email, and SMS) are too invasive or only work with a handful of people before becoming SPAM so I built my own system using a static site generator and an email server. I think it is a novel configuration that may be interesting to people working on self-hosting projects or small group collaborations. Configuration details to follow in later posts.

Social Sharing

I have spent the past couple of weeks wrestling with a seemingly simple question: how do you share photos without using a social network? The recent birth of my son Jasper means that I have a newfound need to share pictures with family and friends but all the existing solutions are either invasive (social networks) or will only work for a handful of participants before turning your happy news into SPAM (email, SMS). Interestingly, I have found a rather large portion of my friends who are fine with social networks in general but not for pictures of their children. I also have a number of friends who use social networks for everything but feel SPAMed by people’s baby pictures there. I set out to find some better options to address this fairly widespread need.

Defining the need

Like any good technologist I started out by defining what I need from this social sharing tool and then turned those goals into technical requirements. The general goal is to have a tool that lets my wife and I privately share photos and text with family, builds an archive or record of what we post, will work with everyone’s software setups, and is easy enough to use that we actually use it. As I worked through some of the details of these goals I distilled them into these requirements:

  1. Must run on servers I can trust.
  2. Must use software people are already using or very general purpose free software.
  3. Must use authentication for reading and posting but must not require me to manage passwords for each user.
  4. Must not flood people with announcements.
  5. Must have browse-able archive.
  6. Must be able to post from and read on iOS devices.
  7. Cannot be harder to add content to than it is to send an email.
  8. The hosting software must require minimal maintenance and have minimal security risk.

The Candidates

With requirements in hand I started looking at the available technology out there for a solution. Five emerged for consideration: commercial social networks (facebook, instagram, etc), free software social networks (diaspora, frendika, etc), email, mailing lists, and blogs.

Commercial social networks

Facebook, Instagram, etc all run on commercial servers designed to datamine your social interactions and profile all participants so they fail criteria 1: run on servers I can trust.

Free software social networks

Diaspora, frendika, etc could be good fits for this situation, especially because, of all the self-hosted options, these offer the easiest fine-grained permission management. Unfortunately no one in my social circle is using them, including myself, so they fail criteria 2: use software people are already using.

Email

Email actually comes pretty close here, which is probably why it is the main tool my friends use for sharing photos of their children with family. Unfortunately, it does flood people so it fails at criteria 4 and it does not provide an archive for recipients so it fails criteria 5. People can build their own archive by saving all their messages but this only works for original recipients, not someone we add later or might have forgeten to include on some individual messages.

Mailing list

Mailing lists actually solve almost all the issues with email. You get a browse-able archive. People can manage their own subscriptions so they can unsubscribe if they are getting too many messages. People do need new passwords but the listserv software manages storing those and has built-in password reset capabilities. Unfortunately, I know many of my family and friends simply get so much email or have too few tools to effectively filter the email they get that even something as easy to process as a mailing list would be burdensome or get ignored. Ultimately, this still fails criteria 4: don’t flood.

Blog

This also hits a number of the big requirements, especially using a static site generator to achieve the low maintenance and security risk of criteria 8, but some challenges remain. The biggest issues are authentication and ease of posting. Since I will be running the web server I can use whatever authentication I wish but I do not want to have to setup and maintain passwords for all of my friends and family. Posting is simply hard, especially from a mobile device, and that is where my custom work comes in.

For authentication I looked into simplifying things with RSS. Many of my family members do use RSS readers for news browsing on their iPads and RSS clients are general enough that I would be comfortable telling family to install one in return for baby photos. I would also feel better about supporting individual passwords for client software than I do for general web pages because I know the websites will be viewed on multiple devices (requiring multiple times the password support). Sadly none of the free software RSS readers for iOS support authenticated feeds so this fails criteria 6: iOS required. I ended up settling on simple authentication with shared credentials for all readers and moving posting authentication to the email submission. If anyone knows of a free software iOS reader that can handle authenticated feeds, I would love to hear about it. The Android spaRSS reader worked wonderfully if anyone is looking.

For ease of posting I had a problem. Logging in and posting to a blog is just not easy enough, especially if you are posting a gallery of photos, which is the main purpose of the project. I knew that if things were more complicated than sending an email we would just end up falling back to sending emails and only our parents would ever hear about Jasper. I started looking at existing email-to-blog tools but all of these either require you to trust your blog software with your email login credentials or trust a third party to process your email and post your updates. I decided that email-to-blog was the right approach but that having my blog check email for me would break criteria 8: server software should have low maintenance and security risk. In the end I decided the only blogging software durable enough for me to configure and then ignore were the static site generators, none of which have an email-to-blog submission tool that I could find. So I needed to build my own.

The Plan

What I ended up designing, and am now almost done building, is a specially configured email server (postfix) that is set to only accept email from a couple of people, to pass those messages to a local script that converts them into blog-formated plain text files with accompanying directories of images, and then feeds that into a static site generator (pelican) sitting on my webserver (Apache). I used whitelists and some anti-SPAM tools on the mail server to control who can post based on their existing email addresses, which means my wife and I can post without needing new passwords or to remember to send the post from special accounts. Using this approach it is possible to turn postfix into a sort of file-based application server that may be useful in a number of situations. I think this approach has particular potential for those looking to build a system somewhere between the simplicity of static site generators and the complexity of dynamic tools like WordPress.

Currently the mail server configuration and the static site are both complete but the script to move the email messages along is still in process. I will write more detail in the next few posts on how I configure both postfix and pelican for this project and what some of the security considerations are for this using the kind of delegated authentication scheme.

Learning in Public

For the last six weeks I have been running an experimental project at Columbia, an online course called “Reclaim a personal space for education online” but which I think of as “Learning in Public”. The course runs on the P2PU platform and is modeled on the Domain of One’s Own initiative, a fantastic effort to teach students how to build and operate their own online spaces using WordPress and other Free Software. I’ve learned a number of lessons from the first half of the course and produced some resources that I hope will be of use to anyone else who is interested in running their own DIY online course.

Moving online

When Domain of One’s Own first launched at the University of Mary Washington (UMW) it was tied to an undergraduate course on digital storytelling that provided scaffolding for learning the mechanics of how to set up WordPress and operate your own internet domain. Since I work primarily with instructors, who are generally unable to fit such a commitment in a full language course load, I designed this course to operate entirely online and replaced the storytelling scaffolding with an exploration of what it means to move your private learning into a public space. As supporting materials I also threw in a strand of topics giving a technological introduction to the web (what are web sites made of, how are they shared, etc). There are never enough opportunities for general technology education and, while we are being reflective about the effects of using technology, it seemed appropriate to support a similar exploration of how that technology actually works, even if only in outline.

Learning in Public

I have felt for a few years now that we need to do a better job teaching people how to navigate and participate effectively in networked communities. Over the years I have talked with many people running Summer of Code, Wikipedia Education, and other similar community internship or engagement projects. Almost all the participants in such programs run into the same troubles identifying and navigating social norms, learning to use the public communication tools that tie such communities together, and publicly sharing their own work in useful ways.

This is perfectly understandable since I know of no concerted effort to teach these skills, which people often seem to assume are natural abilities for “Digital Natives” or are unaware of how real and valuable such skills are. I think this is the most glaring oversight in all the core curricula out there. For that matter, we could probably run a semester-long technology course on “Technology Triage and How To File Useful Bug Reports” to the great benefit of all involved and society at large. This course is not that one, nor does it include most of the material I would include in a a real “Learning in Public” course since the main focus remains on building personal websites. But, at least informally, this is my first attempt to build up and publish some of those materials where I hope they can do some good. If you are interested in weaving some of those materials into your own courses in future, you might take a look at these three of the pieces I’ve put together so far: What is a Website?, Ethnography of an online community, and Copyright Soup: CC, OER, and Bears, oh my!.

Is any of this actually what people want to learn? So far all of the participants have been focused on building their sites but I am not rushing to judgment. This semester’s group is too small a sample size to draw conclusions from (see Advertising). The material I’ve built so far has been nicely reusable in other contexts around the LRC so I consider that a sign that I am on the right track. Whenever initiatives reinforce each other like that it is always a win in my book, which was part of my goal in designing this experimental course out of mutually reinforcing but ultimately independent tracks. If you are putting together your own online course, I would strongly advise a similar reusable module approach.

The Tools

After running Domain of One’s Own for a couple of years at UMW the origianl organizers of the project spun off a hosting company to help other universities manage similar projects and to provide individual hosting geared towards the needs of the educational community. They call that company Reclaim Hosting and I knew they were going to be the perfect fit when I started planning out this course. They have a great combination of strong values, great support, and rock bottom pricing and I am very happy to have gone with Reclaim for this project.

Since a big part of the goal for this course was to expose instructors to the realities of taking a course in public online so the normal university course tools were not going to work. For better or worse all of the big “MOOC” platforms (obligatory reference to moocthulhu) are locked down to particular instructors and institutions so I built the course on P2PU, a post-OpenCourseWare, pre-MOOC open fever open education platform. That had some pros and cons.

P2pu is effectively a public wiki and suffers from all the common signs of wiki graveyard syndrome, lots of half-finished courses or courses that have not ben run in years and where it is unclear how you could get control of the material should you wish to run on yourself. There are also some signs of technological aging on the platform itself that made me wish I were using WordPress instead. From the fact that they have re-designed the main website to focus on how to run an in person learning group while hiding the actual course listings, I think they are aware of the difficulties and are trying to focus on reusing the existing courses that were finished and are useful. Unfortunately, this makes it all but impossible for people to discover new courses on the site, which should be one of the main advantages of using such a platform as opposed to just building things on your own blog.

While I think it is great that p2pu gives people a public platform for building freely licensed course, if I were doing it over again I would just build it in a Reclaim Hosting account. Maybe next time I’ll do that and use it as an opportunity for a BuddyPress component. In the meantime I will be completing the course as initially planned so that I don’t contribute to the wiki graveyard, but as I go I am also migrating all the content over to github at this address: github.com/edtechgarden/edtechgarden.github.io so the materials can be more widely used.

Advertising

P2PU’s advice is right on point here. Organize a local community, even if offering the course publicly, and start doing that organizing a month before you start the course. This is where I need to put the most work next time around. While the course has generated decent interest, people seem very reluctant to join a course already in process, especially people who work at universities. If I had spent more time lining up all the interested parties before starting the sessions, the course would have been larger and the discussions correspondingly more diverse.

Reuse and the future

As I mentioned earlier, I have found a number of ways to reuse much of the content created through this course, whether in individual consultations with instructors, a forthcoming group workshop trying to condense the course into one hopefully memorable afternoon, or as a series of blog posts coming out later this month. I also think some of it may end up in the Reclaim Hosting documentation. One of the things that continues to surprise me, even 13 years after my first free software/free culture experiences, is just how widely things can spread when they are freely licensed. My introductory lessons about WordPress may be useful for Reclaim not because they are the world’s best but because almost all of the copious WordPress documentation online is under a restrictive copyright license and reclaim freely licenses their own documentation.

After this term I want to keep some of the momentum going by collecting other general technology and education material in the same github account. Feedback or contributions are certainly welcome. If you don’t have a github account, you can just send me an email. I’m ian@ this site’s domain name.

Innovation in practice

In each Supreme Court brief that SFLC has filed over the years we have included a little note on the first page declaring that the brief was made using only free software. This point was particularly important in our most recent brief, for a case named Alice Corporation v. CLS Bank, which was argued in front of the court last week. Our use of free software was particularly important this time because we argue in our brief that free software has been responsible for the major software innovations of the modern era. In partial support of that claim I want to show you our document creation process and tell you about the free software we use to take text from an email and turn it into a camera-ready Supreme Court brief, then a website, then an eBook.

1: Markdown – the drafting tool

The first tool we use, markdown, is a standard way to use keyboard characters like *, #, and [ ] to indicate what parts of a text should be italicized or in bold and which parts are things like headers or footnotes. Using standard characters to indicate these formatting options helps keep the focus on the text and makes it possible to use a variety of collaboration tools during drafting like etherpad, wikis, and version control systems.

While we have all of those tools running at our office, and used many of them to share drafts of the brief, the initial versions of the CLS brief began as simple email messages. Markdown is a great tool for sharing documents via email because you can include the text of the message directly in the body of the message making it easy to make edits or add comments just by replying. Keeping the text of a document inside the email message also keeps every version track-able and search-able with your normal email tools. Contrast that to the common practice of sending documents around as attachments and how hard it is to identify which of the dozen files you may have is the first time a particular piece of text was introduced or a particular change was made.

2: Pandoc – the swiss army knife

Our second tool, pandoc, helps us translate the simple formatting and structure marks from our draft into whatever type of more formal document we need. When it comes to conversion, Pandoc is a swiss army knife that can turn our Markdown document into a webpage, word processor document, Wikipedia article, eBook, or many other formats. The format we want is LaTeX, which is a system for laying out complex documents that require precise formatting. It is commonly used for books, résumés, journal articles, dissertations, and, in our case, Supreme Court briefs.

To convert our “draft.mkdn” document to LaTeX we simply run “pandoc draft.mkdn -o draft.tex” which tells pandoc to look at the markdown file and output a LaTeX version of it. Since “.mkdn” and “.tex” are standard extensions for Markdown and LaTeX formatted documents, Pandoc understands the conversion we want it to do without any other instructions. For now that one command is all we want from pandoc but we will come back to it when building our eBook later on.

3: LaTeX, a personal print shop

Our third tool, LaTeX, is the software version of a professional print shop. It has everything you need to format and lay out documents down to the smallest printing measurement. This is different from a word processor in two important ways. The first is that LaTeX has a professional quality typesetting engine that calculates optimal spacing and line breaks. The result is simply beautiful documents. Take a look at this sample text printed as either a LibreOffice word processor document (pdf) or using the default LaTeX settings (pdf). Try printing that text with whatever tools you normally use and see which one you think is more attractive and pleasant to read.

The second important difference between LaTeX and a word processor is that LaTeX keeps the layout and document formatting information separate from the document contents. Importantly for us, this means we can do the complicated work of specifying all the Supreme Court’s detailed document requirements for briefs just once and then reuse that for each brief we file. We can even share that formatting file with you (brief.cls) so you can study how it works or use it for your own Supreme Court brief.

Once our draft text is in LaTeX format, which we get from pandoc, the final rounds of editing and polishing begin. This is the stage where we make sure that all the citations are correctly formatted and everything looks good on paper. For the CLS brief we made a couple tweaks, adding a page break before the last section so the last page would have more than two lines on it and deciding to indent the subheadings in the table of contents by adding a new option to the formatting “class” file.

When everything looks good, LaTeX creates a PDF that we can send directly to the printer for binding and delivery. Because all of the formatting work is done by LaTeX our document is “camera ready” which means we never have to worry about problems like using a different word processor version from the printer. As soon as the document leaves our hands we have complete confidence about how the document will look when printed. If there were any question about pdf compatibility LaTeX could even output a version in printer-native postscript format. As far as the court is concerned, our work is done once we send the file to be printed but we feel it is important to make the work we do as widely available as possible. To that end we try and share our publications in as many formats as possible that people might want. Our next two steps will take the same LaTeX file and turn it into a webpage and then an eBook.

4: Tex4ht

Turning a LaTeX file into a webpage is greatly helped by a tool called tex4ht which is an entire webpage layout engine built for LaTeX. The resulting website code can be a bit unorthodox when preserving some of the print features not used online, for instance small caps are created by formatting each letter separately and matching up the sizes, but it is very effective at transforming even complicated print documents into websites. It is the natural choice for something like a Supreme Court brief. You can see the results of running tex4ht on our final LaTeX brief at our site, which is basically unchanged from the default output of running “tex4ht alice-cls-amicus.tex” except for adding our standard website header and removing some extra dashes on the cover page.

5: Calibre

Transforming our LaTeX brief into an eBook begins with pandoc, the same tool we used to create our first LaTeX draft out of markdown. In addition to working with markdown, pandoc is able to convert to and from a number of other formats, including translating LaTeX into the “epub” eBook format. This is an important feature for us since, however careful our drafting efforts, some changes always creep in once our brief has moved out of markdown and into the review and proof reading passes. Pandoc’s ability to read as well as create LaTeX documents lets us focus on editing the document rather than worry about which format it is in during a particular editing step.

To get a rough version of the ebook we want we just run “pandoc alice-cls-amicus.tex -o Initial-eBook.epub” and pandoc will figure out what format we want from the file extensions. Because pandoc lacks some of the creative formatting tricks that tex4ht uses, like creating a word in small caps by formatting each letter separately, this initial eBook version is missing some of the elaborately formatted parts of the brief like the cover page.

Once we have this initial eBook we can polish up the rough edges using a free eBook library program called Calibre. Adding our draft to the Calibre library gives us access to a whole editing toolkit. Just select the book and hit the big “edit” button in the main toolbar. Since eBooks are basically websites, this final stage of processing is actually simple HTML editing. Conveniently, this means that we can replace the missing cover page by copying and pasting that portion from our website version. Since pandoc preserved all of the in-text formatting like italics and long dashes, all that is left to do is make sure that the sections and subsections all have appropriate HTML heading numbers, add in the spacing dots on the table of authorities, and tell Calibre to generate a table of contents.

This was actually my first time editing an eBook so that whole process tool me about an hour.

Conclusions

If you have only ever produced documents using commercial word processing software, many of the tools I have described here may seem strange to you. Like so much of Free and Open Source Software, the tools we use for making documents come from a variety of different authors who each produce documents their in own ways. The particular tools we use at SFLC have been chosen over the years as our practice grows and evolves. Whether you are looking to replace your document tools entirely, try out a new way to turn emailed text into word processor documents, or just see what it looks like to take free tools and put them together into a system that fits your office, I hope this has been an informative look at some innovative free software.

(Originally posted 2014-04-11 on the Software Freedom Law Center’s Blog)

css.php