Back in 1974, when the Internet was a fraction of what it is now, the acorn to an oak, there were really only two applications,” said Shirky, “Telnet, and FTP.”

Surely he’s wrong, I thought.  Those protocols aren’t that old.

But I was wrong. FTP was invented in 1971, and telnet was developed in 1969.

(Telnet is a way to connect interactively with another computer. In practice it’s been replaced by the more secure ssh, but vestigial copies remain on all modern computers.)

What really threw me wasn’t that telnet was from 1969 as much as that it was RFC 15.  In the networked world, Requests for Comments are documents which define the standards that computers use when communicating with each other.  To understand how old RFC 15 is, consider that the venerable FTP is RFC 114,  while email as we know it is RFC 821 (1982), and HTTP is RFC 1945 (1996, although obviously it had been in use for years). The most recent RFC is 5382. RFC 15 is ancient history.

Because I am a nerd I spent some time browsing the early RFCs, and I was struck by how charmingly antique they are. RFC 16 says that M.I.T. should receive copies of RFCs. RFC 6 begins, “I talked with Bob Kahn at BB&N yesterday.” RFC 14 never existed.

RFC 7 (“Host-IMP Interface”) includes a prefatory note:

The original of RFC 7 was hand-written, and only partially illegible [sic]
copies exist.

Indeed, the actual RFC begins:

This paper is concerned with the preliminary software design of the
Host IMP interface. Its main purpose is on the one hand to define
functions that will be implemented, and on the other hand to provide
a base for discussions and …(unreadable).

I’m on the mailing list for users of the Text Encoding Initiative (TEI), an XML schema used primarily for encoding historical texts. The schema is equipped with tags for tracking everything about a document, including changes that occur over centuries of time. On the TEI list, people ask questions like, “How do I represent a medieval manuscript and also indicate which passages were underlined by an 18th century owner?” or “What tag should I use for a poem title that was handwritten vertically in the left margin?” (Promptly followed by vigorous scholarly debates over the “correct” answers.)

There’s something charming about how early internet history, just 40 years old, is almost as poorly documented and in need of careful archivists.

The source documents are encoded in TEI XML and rendered to the browser using Python and lxml, three of my favorite technologies.

There are a few take-aways from this project that might benefit anyone working in a similar area and scale:

• Use a standard encoding format (in this case TEI, but choose an appropriate one based on the source content)
• Use a modern programming language, even in a humanities context (e.g. Python)
• Use modern XML parsing tools (e.g. lxml + XPath + XSLT)

The key advantage of libraries such as lxml in publishing and digitization projects is that it allows the developer to freely mix XML-native languages like XPath and XSLT with the expressive, procedural programming style of Python. I’m still amazed by how many people are “parsing” XML using regular expressions (or worse), or using plain CGI/Perl scripts to serve up content. There are easier ways!

“Free” doesn’t have to mean primitive. In fact I would argue that projects like Pinax can jump-start library or digital archive sites into the 21st century with less work than a grad student will spend crafting a bespoke Perl script.

Congratulations to Thomas Crombez and his team!

To expand the project in useful ways I’d like to be able to add:

1. Other content types besides novels, especially reference
2. Content from other document formats, such as DocBook
3. Native, highly-tagged TEI documents

Wikipedia and its cohorts are by far the largest source of public domain data on the web now, but they aren’t encoded in XML. Publishers are unlikely to use wiki formatting to mark up their content and thus developing a workflow to convert from wiki to TEI doesn’t seem productive.

XML data welcome!

TEI is the Text Encoding Initiative, and is one of the most popular markup formats for printed works (especially in academics). All of the content on threepress has been converted from the Gutenberg format to TEI upon ingestion into the site.

epub is the shorthand for the e-book format proposed by the International Digital Publishing Forum (IDPF), which uses XHTML and custom metadata formats. An e-book bundle is distributed in ZIP file format with its text and supplementary media “bound” together.

tei2epub is written in Python with XSLT. It also comes bundled with the latest version of epubcheck, for validating the output. It is meant to be used by developers rather than end-users (unlike the recent BookGlutton epub converter) and as most of the functionality is in the XSLT, should be easy to port to other languages. Like all threepress tools it is released under the BSD license which means it is free for all commercial and non-commercial use. You may download the ZIP version of the current release or get the latest version from svn at http://epub-tools.googlecode.com/svn/trunk/

Current limitations:

1. tei2epub has not been tested on extensively marked-up TEI. It leverages the standard TEI to XHTML stylesheets distributed by TEI, but it is unknown whether epub readers will support all of the resulting markup
2. It accepts only a single source document (i.e. an entire TEI book)
3. It does not handle images or other kinds of media

Any of the above can be addressed with the addition of more complex TEI source books.

Edited May 22, 2008 to point resources at a new standalone repository.