The Humanities E-Book (HEB) project took 20 books (as scanned page images + uncorrected OCR) and converted them to an in-house XML format. They compared the workflow impact, costs and user experience of the final XML product versus that of the page-image ebooks.

Many of their findings consorted with my own experience in this area:

1. The quality of the OCRed text was worse than expected: good enough for search, but not always suitable for reading.  However, the cost of double-keying the text from scratch was prohibitive.
2. The encoding vendor, while skilled and diligent, nevertheless produced output that would require a trained editor to correct properly.  HEB spent 4-8 hours hand-correcting each book in the sample set.
3. The average cost for conversion to XML was approximately 3X greater than for scans + OCR only.  This did not include in-house correction and review.

User survey results

After the 20 sample books were made available to their community, users were polled for their reactions.  I feel these are worth mentioning at length.

69% of readers preferred the XML-encoded books (presented as HTML in a browser).

Reasons for preferring the XML scans included:

1. Readability (despite the fact that not all books were completely proofed)
2. Usability (e.g. cut and paste, ability to use screen readers)
3. Layout (the HTML presentation had few distracting elements on the pages, and more content was available per web page than in the page-based scans)

Interestingly, of those readers who preferred the image scans, one of the primary reasons cited was the more book-like paginated layout. I’m very conscious of this tension: many Bookworm users complain about the chapter-at-a-time, scrolling layout of the pages, while others absolutely hate arbitrary emulation of the printed work. It seems to be a strong personal preference that runs in one direction or the other.

Ebook interface considerations

Although not directly related to the study at hand, I found some of the publisher-imposed constraints on their user interface illustrative. I feel these would be best be avoided when designing an ebook reading site:

Foremost among user requests was a desire for better printing options. Printing of HEB titles has always been restricted to fair-use provisions, and for this reason there had neverbeen any immediate way of printing out pages without prior browser adjustment to
accommodate frames—the intention being to discourage printing out long sections of copyrighted text at once.

The ability to print text at length is critical for any serious work.  I’m always unhappy when a site prevents me from doing an ordinary task like printing or downloading.  I hope that publishers reconsider these types of restrictions.

Similarly, revenue models should not constrain the ways in which licensed users can access content:

XML titles normally suppress all higher-level “container” sections, so that users always access only the smallest available text chunk in each overarching section. [...]

…for this set of titles, we would simply make all section levels accessible.  This would affect the process of tallying hits for these titles—something needed in order to calculate royalties  for publishers and usage statistics for libraries—as users could now potentially read an entire book by accessing only a small number of chapter-level sections (which in turn would generate fewer hits than reading the page-image version).

As a reader, I should absolutely be able to read content — especially XML-based content — in as fluid a manner as possible.  Generating accurate accounting is a programming problem, and not one that should drive decisions about the reading interface.

Thanks to everyone who attended and especially to those who asked so many excellent questions.

Also on that page at the moment is a giant photo of my head advertising What Publishers Need to Know About Digitization, a web seminar I’ll be hosting with O’Reilly Media on November 12. It will be a very high-level, introductory overview aimed at non-technical staff in publishing who are considering a digitization project.

Going full-circle, I wonder if there would be interest in a simple web-based OCR service where publishers could upload a scanned document to see how well bare-bones OCR performed on an image-only PDF or JPEG scan. I imagine it might help predict the complexity of a digitization project, and understand some of the challenges inherent in the process.

Back in 1974, when the Internet was a fraction of what it is now, the acorn to an oak, there were really only two applications,” said Shirky, “Telnet, and FTP.”

Surely he’s wrong, I thought.  Those protocols aren’t that old.

But I was wrong. FTP was invented in 1971, and telnet was developed in 1969.

(Telnet is a way to connect interactively with another computer. In practice it’s been replaced by the more secure ssh, but vestigial copies remain on all modern computers.)

What really threw me wasn’t that telnet was from 1969 as much as that it was RFC 15.  In the networked world, Requests for Comments are documents which define the standards that computers use when communicating with each other.  To understand how old RFC 15 is, consider that the venerable FTP is RFC 114,  while email as we know it is RFC 821 (1982), and HTTP is RFC 1945 (1996, although obviously it had been in use for years). The most recent RFC is 5382. RFC 15 is ancient history.

Because I am a nerd I spent some time browsing the early RFCs, and I was struck by how charmingly antique they are. RFC 16 says that M.I.T. should receive copies of RFCs. RFC 6 begins, “I talked with Bob Kahn at BB&N yesterday.” RFC 14 never existed.

RFC 7 (“Host-IMP Interface”) includes a prefatory note:

The original of RFC 7 was hand-written, and only partially illegible [sic]
copies exist.

Indeed, the actual RFC begins:

This paper is concerned with the preliminary software design of the
Host IMP interface. Its main purpose is on the one hand to define
functions that will be implemented, and on the other hand to provide
a base for discussions and …(unreadable).

I’m on the mailing list for users of the Text Encoding Initiative (TEI), an XML schema used primarily for encoding historical texts. The schema is equipped with tags for tracking everything about a document, including changes that occur over centuries of time. On the TEI list, people ask questions like, “How do I represent a medieval manuscript and also indicate which passages were underlined by an 18th century owner?” or “What tag should I use for a poem title that was handwritten vertically in the left margin?” (Promptly followed by vigorous scholarly debates over the “correct” answers.)

There’s something charming about how early internet history, just 40 years old, is almost as poorly documented and in need of careful archivists.

The source documents are encoded in TEI XML and rendered to the browser using Python and lxml, three of my favorite technologies.

There are a few take-aways from this project that might benefit anyone working in a similar area and scale:

• Use a standard encoding format (in this case TEI, but choose an appropriate one based on the source content)
• Use a modern programming language, even in a humanities context (e.g. Python)
• Use modern XML parsing tools (e.g. lxml + XPath + XSLT)

The key advantage of libraries such as lxml in publishing and digitization projects is that it allows the developer to freely mix XML-native languages like XPath and XSLT with the expressive, procedural programming style of Python. I’m still amazed by how many people are “parsing” XML using regular expressions (or worse), or using plain CGI/Perl scripts to serve up content. There are easier ways!

“Free” doesn’t have to mean primitive. In fact I would argue that projects like Pinax can jump-start library or digital archive sites into the 21st century with less work than a grad student will spend crafting a bespoke Perl script.

Congratulations to Thomas Crombez and his team!