The Humanities E-Book (HEB) project took 20 books (as scanned page images + uncorrected OCR) and converted them to an in-house XML format. They compared the workflow impact, costs and user experience of the final XML product versus that of the page-image ebooks.

Many of their findings consorted with my own experience in this area:

1. The quality of the OCRed text was worse than expected: good enough for search, but not always suitable for reading.  However, the cost of double-keying the text from scratch was prohibitive.
2. The encoding vendor, while skilled and diligent, nevertheless produced output that would require a trained editor to correct properly.  HEB spent 4-8 hours hand-correcting each book in the sample set.
3. The average cost for conversion to XML was approximately 3X greater than for scans + OCR only.  This did not include in-house correction and review.

User survey results

After the 20 sample books were made available to their community, users were polled for their reactions.  I feel these are worth mentioning at length.

69% of readers preferred the XML-encoded books (presented as HTML in a browser).

Reasons for preferring the XML scans included:

1. Readability (despite the fact that not all books were completely proofed)
2. Usability (e.g. cut and paste, ability to use screen readers)
3. Layout (the HTML presentation had few distracting elements on the pages, and more content was available per web page than in the page-based scans)

Interestingly, of those readers who preferred the image scans, one of the primary reasons cited was the more book-like paginated layout. I’m very conscious of this tension: many Bookworm users complain about the chapter-at-a-time, scrolling layout of the pages, while others absolutely hate arbitrary emulation of the printed work. It seems to be a strong personal preference that runs in one direction or the other.

Ebook interface considerations

Although not directly related to the study at hand, I found some of the publisher-imposed constraints on their user interface illustrative. I feel these would be best be avoided when designing an ebook reading site:

Foremost among user requests was a desire for better printing options. Printing of HEB titles has always been restricted to fair-use provisions, and for this reason there had neverbeen any immediate way of printing out pages without prior browser adjustment to
accommodate frames—the intention being to discourage printing out long sections of copyrighted text at once.

The ability to print text at length is critical for any serious work.  I’m always unhappy when a site prevents me from doing an ordinary task like printing or downloading.  I hope that publishers reconsider these types of restrictions.

Similarly, revenue models should not constrain the ways in which licensed users can access content:

XML titles normally suppress all higher-level “container” sections, so that users always access only the smallest available text chunk in each overarching section. [...]

…for this set of titles, we would simply make all section levels accessible.  This would affect the process of tallying hits for these titles—something needed in order to calculate royalties  for publishers and usage statistics for libraries—as users could now potentially read an entire book by accessing only a small number of chapter-level sections (which in turn would generate fewer hits than reading the page-image version).

As a reader, I should absolutely be able to read content — especially XML-based content — in as fluid a manner as possible.  Generating accurate accounting is a programming problem, and not one that should drive decisions about the reading interface.

Thanks to everyone who attended and especially to those who asked so many excellent questions.

Massive XML files are uncommon in the general web development world, where the primary roles of XML are either as configuration files, read only infrequently, or for interchange across the web, in which case the files are necessarily small. It’s rare to encounter XML measured in gigabytes or more; data at that level is usually stored in a relational database.

For that reason I find myself frustrated with many XML tools, even those ostensibly designed to handle large amounts of data.  Too often they don’t scale well or at least easily.  I don’t believe that scaling should be a black art that each individual developer needs to solve independently.  Unfortunately, in commercial products ease-of-use is a key bullet point and computationally-difficult problems are hard to summarize in a user’s guide.

I tend to recommend open-source software most strongly in two scenarios: for small projects with limited budgets and for large projects with unique challenges.  There simply isn’t going to be a one-size-fits-all application for most interesting publishing work.

This is one of many reasons I’m excited by Google’s willingness to open its Google Books archive to researchers:  Python is a first-class programming language in the Google ecosystem, and Google has a good track record of open-sourcing those internal tools with limited commercial value.  I expect a lot of interesting work to come out of that archive once it’s available.

Also on that page at the moment is a giant photo of my head advertising What Publishers Need to Know About Digitization, a web seminar I’ll be hosting with O’Reilly Media on November 12. It will be a very high-level, introductory overview aimed at non-technical staff in publishing who are considering a digitization project.

Going full-circle, I wonder if there would be interest in a simple web-based OCR service where publishers could upload a scanned document to see how well bare-bones OCR performed on an image-only PDF or JPEG scan. I imagine it might help predict the complexity of a digitization project, and understand some of the challenges inherent in the process.

The source documents are encoded in TEI XML and rendered to the browser using Python and lxml, three of my favorite technologies.

There are a few take-aways from this project that might benefit anyone working in a similar area and scale:

• Use a standard encoding format (in this case TEI, but choose an appropriate one based on the source content)
• Use a modern programming language, even in a humanities context (e.g. Python)
• Use modern XML parsing tools (e.g. lxml + XPath + XSLT)

The key advantage of libraries such as lxml in publishing and digitization projects is that it allows the developer to freely mix XML-native languages like XPath and XSLT with the expressive, procedural programming style of Python. I’m still amazed by how many people are “parsing” XML using regular expressions (or worse), or using plain CGI/Perl scripts to serve up content. There are easier ways!

“Free” doesn’t have to mean primitive. In fact I would argue that projects like Pinax can jump-start library or digital archive sites into the 21st century with less work than a grad student will spend crafting a bespoke Perl script.

Congratulations to Thomas Crombez and his team!

This is the output from my Google Analytics web traffic report on the country which sends the most visits to threepress.org. 44% of the traffic to the entire site, which includes this blog, some public domain ebooks and my contact information, is to the ePub validation service, a wrapper around Adobe’s epubcheck. (Bookworm statistics are not included in this report.)

India sends three times as much the traffic to the validation page compared to second-place United States, but only one-third as much the traffic to the home page.

It’s even more interesting to look at the “bounce rate” for the home page by country. The “bounce rate” is the percentage of times that a given page is the last one that a user looks at before they leave the site, and it’s one of the most useful metrics in web analysis. The overall bounce rate for the threepress.org home page is 37%, meaning 37% of the people who visited that page didn’t have a reason to click on another link. For India, that figure is 5% — presumably because they are all clicking through to the validation service. (By contrast, 80% of South African visitors leave immediately, suggesting that some unrelated keyword searches or links are driving them there.)

So if you’re looking for vendors who can provide high-quality, valid ePubs, I’d suggest, in descending order of frequency, suppliers in these cities:

1. Pune
2. Delhi
3. New Delhi
4. Chennai
5. Mahape