Though I was the principal author of the paper, I’d like to thank the other members of the JISC team for their support, edits, and encouragement: Theo Andrew, Peter Sefton, Emma Tonkin, and David Flanders. Many other deliverables were produced as a part of this project, including digital publishing toolkits and usability studies; please check them out on the jiscPUB blog.

Executive summary

…This report aims to describe some historical perspective on electronic publishing, leading up to why the “ebook revolution” has happened in the 2010s when it had failed to take hold before. We will describe some of the details of how digital books are authored, both in a scholarly context and in general ebook production terms. We have included in-depth coverage of the unexpected outcomes of ebook distribution, including issues of rights, royalties, copyright, academic impact, and the implications of limiting access to and reproduction of digital books. Finally, we review findings from a number of ebook pilot programs conducted in U.S. universities, and draw from the work done by the rest of the “JISCpub” team in uncovering possible future work that could be actionable and relevant to a scholarly publishing audience, with a goal towards providing better tooling for both authors and readers of scholarly works.

Recommendations in brief

1. Rich full-text semantic search tools for scholarly ebook collections.
2. Tools for generating or traversing highly-specific stable citations.
3. Development of a pilot to produce student theses with high-engagement linked-data content.
4. Plugins or add-ons to provide simple, ebook output for popular word processing tools.
5. Improved workflows for authoring attractive, accessible, standards-based mathematical notation in ebooks.
6. Development of an ereading system with an emphasis on scholarly annotation and research-gathering.
7. Provisions to train and share scholars interested in digital publishing.
8. Aggregate ebook services for authors and university presses.
9. Maximize use of orphan works.
10. Community resources for institutions with digital collection.

Read the complete report.

For skilled cataloging, work that takes intelligence and the best education attainable, the demand is always above the supply. In the library of the future, the free library, the greatest missionary force of the age, there is going to be a great opening for women’s work, too.

I don’t mean by the ideal free library a big central book collection in this city or any city, but smaller ones, so scattered as to bring the reading-room within half a mile of every workman’s home.

–”Facts for the Fair Sex, Some Things that Women Can Do Better Than Men”, Chicago Tribune, May 1887

The Humanities E-Book (HEB) project took 20 books (as scanned page images + uncorrected OCR) and converted them to an in-house XML format. They compared the workflow impact, costs and user experience of the final XML product versus that of the page-image ebooks.

Many of their findings consorted with my own experience in this area:

1. The quality of the OCRed text was worse than expected: good enough for search, but not always suitable for reading.  However, the cost of double-keying the text from scratch was prohibitive.
2. The encoding vendor, while skilled and diligent, nevertheless produced output that would require a trained editor to correct properly.  HEB spent 4-8 hours hand-correcting each book in the sample set.
3. The average cost for conversion to XML was approximately 3X greater than for scans + OCR only.  This did not include in-house correction and review.

User survey results

After the 20 sample books were made available to their community, users were polled for their reactions.  I feel these are worth mentioning at length.

69% of readers preferred the XML-encoded books (presented as HTML in a browser).

Reasons for preferring the XML scans included:

1. Readability (despite the fact that not all books were completely proofed)
2. Usability (e.g. cut and paste, ability to use screen readers)
3. Layout (the HTML presentation had few distracting elements on the pages, and more content was available per web page than in the page-based scans)

Interestingly, of those readers who preferred the image scans, one of the primary reasons cited was the more book-like paginated layout. I’m very conscious of this tension: many Bookworm users complain about the chapter-at-a-time, scrolling layout of the pages, while others absolutely hate arbitrary emulation of the printed work. It seems to be a strong personal preference that runs in one direction or the other.

Ebook interface considerations

Although not directly related to the study at hand, I found some of the publisher-imposed constraints on their user interface illustrative. I feel these would be best be avoided when designing an ebook reading site:

Foremost among user requests was a desire for better printing options. Printing of HEB titles has always been restricted to fair-use provisions, and for this reason there had neverbeen any immediate way of printing out pages without prior browser adjustment to
accommodate frames—the intention being to discourage printing out long sections of copyrighted text at once.

The ability to print text at length is critical for any serious work.  I’m always unhappy when a site prevents me from doing an ordinary task like printing or downloading.  I hope that publishers reconsider these types of restrictions.

Similarly, revenue models should not constrain the ways in which licensed users can access content:

XML titles normally suppress all higher-level “container” sections, so that users always access only the smallest available text chunk in each overarching section. [...]

…for this set of titles, we would simply make all section levels accessible.  This would affect the process of tallying hits for these titles—something needed in order to calculate royalties  for publishers and usage statistics for libraries—as users could now potentially read an entire book by accessing only a small number of chapter-level sections (which in turn would generate fewer hits than reading the page-image version).

As a reader, I should absolutely be able to read content — especially XML-based content — in as fluid a manner as possible.  Generating accurate accounting is a programming problem, and not one that should drive decisions about the reading interface.

The event was hosted by the Boston Public Library. We were given a fascinating tour of the Open Library scanning facility: ten “scribe” workstations, manned by full-time staffers. I didn’t have my camera (and presumably the scanners’ Canon 5Ds were off-limits), but there’s a good report on the facility from Everything Miscellaneous. As developers we were encouraged to build on top of the free content that’s pouring out of the project, and I’ll definitely be giving that some thought.

I haven’t worked directly with or for libraries yet, so for me the day was mostly a note-taking and networking experience. There are a number of technologies I need to learn more about, including the LibraryThing APIs, COinS, unAPI. Bookworm needs to export more structured data in a useful way, especially when it comes to playing nicely with citation management software. I think I now know enough to stay away from XForms.

I’ll definitely be at the code4lib 2009 conference. I even have a vague memory of promising to give a lightning talk on ePub.

Back in 1974, when the Internet was a fraction of what it is now, the acorn to an oak, there were really only two applications,” said Shirky, “Telnet, and FTP.”

Surely he’s wrong, I thought.  Those protocols aren’t that old.

But I was wrong. FTP was invented in 1971, and telnet was developed in 1969.

(Telnet is a way to connect interactively with another computer. In practice it’s been replaced by the more secure ssh, but vestigial copies remain on all modern computers.)

What really threw me wasn’t that telnet was from 1969 as much as that it was RFC 15.  In the networked world, Requests for Comments are documents which define the standards that computers use when communicating with each other.  To understand how old RFC 15 is, consider that the venerable FTP is RFC 114,  while email as we know it is RFC 821 (1982), and HTTP is RFC 1945 (1996, although obviously it had been in use for years). The most recent RFC is 5382. RFC 15 is ancient history.

Because I am a nerd I spent some time browsing the early RFCs, and I was struck by how charmingly antique they are. RFC 16 says that M.I.T. should receive copies of RFCs. RFC 6 begins, “I talked with Bob Kahn at BB&N yesterday.” RFC 14 never existed.

RFC 7 (“Host-IMP Interface”) includes a prefatory note:

The original of RFC 7 was hand-written, and only partially illegible [sic]
copies exist.

Indeed, the actual RFC begins:

This paper is concerned with the preliminary software design of the
Host IMP interface. Its main purpose is on the one hand to define
functions that will be implemented, and on the other hand to provide
a base for discussions and …(unreadable).

I’m on the mailing list for users of the Text Encoding Initiative (TEI), an XML schema used primarily for encoding historical texts. The schema is equipped with tags for tracking everything about a document, including changes that occur over centuries of time. On the TEI list, people ask questions like, “How do I represent a medieval manuscript and also indicate which passages were underlined by an 18th century owner?” or “What tag should I use for a poem title that was handwritten vertically in the left margin?” (Promptly followed by vigorous scholarly debates over the “correct” answers.)

There’s something charming about how early internet history, just 40 years old, is almost as poorly documented and in need of careful archivists.

The source documents are encoded in TEI XML and rendered to the browser using Python and lxml, three of my favorite technologies.

There are a few take-aways from this project that might benefit anyone working in a similar area and scale:

• Use a standard encoding format (in this case TEI, but choose an appropriate one based on the source content)
• Use a modern programming language, even in a humanities context (e.g. Python)
• Use modern XML parsing tools (e.g. lxml + XPath + XSLT)

The key advantage of libraries such as lxml in publishing and digitization projects is that it allows the developer to freely mix XML-native languages like XPath and XSLT with the expressive, procedural programming style of Python. I’m still amazed by how many people are “parsing” XML using regular expressions (or worse), or using plain CGI/Perl scripts to serve up content. There are easier ways!

“Free” doesn’t have to mean primitive. In fact I would argue that projects like Pinax can jump-start library or digital archive sites into the 21st century with less work than a grad student will spend crafting a bespoke Perl script.

Congratulations to Thomas Crombez and his team!

User-generated content

Tim Spalding from LibraryThing convincingly demonstrated that ordinary readers can, in aggregate, contribute accurate metadata and even scholarly initiatives.  UGC initiatives don’t replace professional cataloging or research, but they can galvanize interest in a subject by using tools “where people are” on the net, whether it’s LibraryThing, Facebook or Amazon.

Once a resource’s content and metadata are available on the net, the library can broaden the scope of what its “local community” means. It may no longer service just people living in its immediate vicinity but anyone who has a interest in the library’s holdings, e.g. retirees who grew up in that area, or individuals with historical interest in the location.

This Flickr photostream from the Library of Congress allows anyone to add historical notes and corrections. Ironically, this project also validates the need for editorial control, as some popular photos are overloaded with inane comments. A sensible moderation policy admits potentially-useful information while deleting random valueless statements (“nice hat!”).

For better or worse, most archival library holdings will draw less attention, and thus UGC is likely to be of higher quality.  Without UGC many collections might languish unseen for decades because the resources don’t exist to professionally catalog them.

Software services and discovery

I’m reading about a book on the net, and decide I’m interested in it — but not to buy.  Perhaps it’s out of print, or extremely expensive, or I’m only mildly curious about the title.  I should be a maximum of one or two clicks away from finding out that it’s available via my local library and ordering it.

(I don’t especially care where the book is or how the library acquires it, although I do need to know an estimated time of arrival in case that’s important to my use case. One ALA speaker suggested the unorthodox practice of buying used books online and mailing them directly to patrons, simply because it can be cheaper than old-fashioned inter-library loan.)

Right now, my local library catalog accepts only inbound requests.  I have to go to the site and initiate a search for the title of interest (assuming I even know what I want). My library network (a consortia of many city libraries in a well-off, highly-educated region) isn’t part of WorldCat and certainly doesn’t provide any advanced discovery tools of its own.

Libraries need to move in the true Web 2.0 direction of providing outbound services.  They should broadcast their catalogs using a simple REST-like API.  It could be as simple as asking for http://mylibrary.org/isbn/123456789 and getting a brief XML response back: the book is available via loan and will take 3-5 days to arrive at the local branch. (An authenticated POST request could then reserve it.) There are already good models for these services in the form of the Google Books and Amazon APIs and there is nothing technically infeasible about it.

The regional library of the future should not be just a physical building to store books but a public service for getting books into its community.

Inspired by the conference, I did come back and make my first online request to my local library. It wasn’t difficult, and this morning I got an email notice that the book is waiting for me at the regional branch a couple blocks away. But it could be even easier, and I’d love to help build out that infrastructure.