The following information is derived from the dc:language field in the OPF file.

Here’s the chart:

Missing from the chart, of course, is English. It’s so overrepresented it skews the chart to the point of being unreadable.

Of the 62,000 epubs on Bookworm right now:

• 29,642 have no language value
• A little over 20,000 are English (combining various values like “en”, “en-GB”, or — embarrassingly — “American”)
• The remainder, 5,874, are distributed among all other languages
• Almost half of the values are represented just one time (likely bad data)

I found it very interesting that the most represented non-English language code is cs — Czech — by a huge margin. Any ideas why?

Wondering which values are correct? The OPF 2.0 spec is unambiguous:

The content of this element [dc:language] must comply with RFC 3066

(Also, does anyone speak “Robert”?)

In ePub, you can’t.

I was surprised to discover that the subset of CSS supported by ePub only includes the rtl or ltr values in the CSS 2 direction property. Rendering vertical text is covered in the CSS3 Text Layout module, which is still in the editorial draft stage. But it does have some unusual browser support already.

There are multiple methods to specify directionality, but the simplest is the shorthand writing-mode property:

writing-mode: tb-rl; /* Chinese */
writing-mode: lr-tb; /* English */

Curiously, writing-mode is only supported by one browser: Internet Explorer. I doubt any other ePub rendering systems besides Bookworm viewed with IE will actually display such content properly (nor are they required to, as writing-mode isn’t in the specification).

Here’s a modified Project Gutenberg book with the English boilerplate and Chinese content rendered with different writing-mode values, as shown above:

I’d like to see future revisions of ePub specifically support writing-mode, even if the CSS Text Layout module isn’t fully ratified at that time. (I may add an issue to the EPUB Maintenance list if that seems warranted.)

Early in Bookworm’s history we didn’t specifically extract the identifier from the ePub and put it in the database, but we do now, so I can query the state of identifiers in the more recent books.

18,571 epubs in Bookworm have some kind of identifier in the database. Many of these were auto-generated by Bookworm, for example if the book had none at all.

14,735 of these are unique. Most of the duplicates came from earlier versions of Calibre, which simply generated identifiers like “25.” Others are just duplicate copies of the same book, which of course is okay.

There are three truly useful identifier types. I’ll specify these in order of my personal preference:

ISBN

Modern commercial books already have unique identifiers. Convenient! Use the ISBN as your primary ID in the epub if you have it, and be sure to use the correct ISBN for your ebook edition:

<dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/"
                 id="bookid"
                 opf:scheme="ISBN">urn:isbn:9780596158347</dc:identifier>

Including the scheme “isbn” in the identifier as well as using the optional opf:scheme attribute allows intelligent reading systems to leverage the ISBN by searching other systems. (1,132 books in Bookworm are courteous in this way.)

URI

Project Gutenberg epubs use a URI. This is a great method for digital-native books with their own steady identifiers. In this case Gutenberg identifiers include the Gutenberg book id:

    <dc:identifier opf:scheme="URI" id="etextno">http://www.gutenberg.org/ebooks/11</dc:identifier>

Like ISBN, these identifiers are stable, and let reading systems do more with the book if they’re able (for example direct users to the canonical information page for that book). If all publishers had their own websites with a page for every book I’d even prefer URIs to ISBNs for this reason.

939 books on Bookworm are from Gutenberg directly, based on searching for this style of identifier. Only about 100 other books use URIs as identifiers.

UUID

A Universally Unique Identifier solves the “required unique” problem by specifying very large numbers, using a variety of schemes.

It’s trivial to generate a UUID in most programming languages. Here’s Python:

>>> import uuid
>>> uuid.uuid4()
UUID('58dce2ac-7aec-45c3-a6de-903a30061545')

Wikipedia lists UUID implementations in other programming languages. You can even just go to a UUID-generating web site. These days, if Bookworm gets an epub with no identifier, it generates a UUID.

UUID provides the least useful information of the three identifier types, but, as Wikipedia points out, “One’s annual risk of being hit by a meteorite is estimated to be one chance in 17 billion [...], equivalent to the odds of creating a few tens of trillions of UUIDs in a year and having one duplicate.” So at least it’s unique.

Smallest 1.6 kilobytes
Largest 233 megabytes
Total # 35,854
Total size 20 gigabytes

I did a frequency analysis of all the individual sizes across the entire corpus:

And then zoomed in on that huge spike in the middle range around 1M and 5M:

So there’s a peak at 3M but really anywhere between 1M and 4M is about average.

I did this analysis earlier in the year when Bookworm had only a paltry 7,800 books, and while the 3M median held, I can say that ebooks in general have gotten larger on average. I attribute that to an increased number of commercially-produced books which include images.

For reference, the Bookworm API documentation is partially reproduced below:

Authorization

You need to sign up for Bookworm before using the API, as access is limited to existing users.

After logging in, visit your Profile to find your api_key. This api_key is a required parameter for all API requests and each request will be authorized against the api_key.

Keep your API key private, like a password. Your API key will change if you update your password.

The API is only available over HTTPS.

Library listing

An API client may request the list of titles in their Library using an authorized HTTP GET.

The results will be returned as an XHTML page that includes every title in one ordered list. Each list item will include a hyperlink to a document address, the Title, Author(s), and Date added. The list will be ordered by Date added. The list will not be paginated. Here’s a sample:

$ curl -ik "https://bookworm_api_test/api/documents/?api_key=your_api_key_here"

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
...

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8" ></meta>
    <title>Bookworm API title list</title>
  </head>

  <body>
    <div id="content">

  <!-- Document list -->
    <ol>

      <li>
        <span class="document-title">Painting the Web</span>

        <span class="document-authors">Shelley Powers</span>
        <span class="document-date-added">2008-07-10 16:17:56</span>
        <span class="document-address">
          <a href="https://bookworm_api_test/api/documents/8/">

https://bookworm_api_test/api/documents/8/</a></span>

        <!-- Painting the Web by Shelley Powers (book.epub) -->
      </li>

      <li>
        <span class="document-title">iPhone: The Missing Manual</span>
        <span class="document-authors">David Pogue</span>

        <span class="document-date-added">2008-07-31 10:34:49</span>
        <span class="document-address">
          <a href="https://bookworm_api_test/api/documents/97/">

https://bookworm_api_test/api/documents/97/</a></span>

        <!-- iPhone: The Missing Manual by David Pogue (9780596153960.epub) -->

      </li>
      <li>
        <span class="document-title">The Way We live Now</span>
        <span class="document-authors">Anthony Trollope</span>
        <span class="document-date-added">2009-03-30 12:36:04</span>

        <span class="document-address">
          <a href="https://bookworm_api_test/api/documents/15215/">

https://bookworm_api_test/api/documents/15215/</a></span>

        <!-- The Way We live Now by Anthony Trollope (Penguin_Trollope_WayWeLiveNow.epub) -->
      </li>

      <li>
        <span class="document-title">The Twitter Book</span>
        <span class="document-authors">Tim O&#39;Reilly, Sarah Milstein</span>
        <span class="document-date-added">2009-04-27 09:21:42</span>

        <span class="document-address">
          <a href="https://bookworm_api_test/api/documents/20258/">

https://bookworm_api_test/api/documents/20258/</a></span>

        <!-- The Twitter Book by Tim O&#39;Reilly... (conv_house_source.epub) -->
      </li>

      <li>
        <span class="document-title">Unit Test: stamp.001</span>
        <span class="document-authors">Norman Walsh</span>
        <span class="document-date-added">2009-05-01 23:43:59</span>

        <span class="document-address">
          <a href="https://bookworm_api_test/api/documents/20941/">

https://bookworm_api_test/api/documents/20941/</a></span>

        <!-- Unit Test: stamp.001 by Norman Walsh (sample.epub) -->
      </li>

    </ol>

    </div>
  </body>
</html>

The listing like the above is available via an authorized HTTP GET from /api/documents/.

Adding titles to your Library

A new title in EPUB format may be added to your Library using an authorized HTTP POST using either of two techniques: directly via EPUB upload or indirectly by EPUB URL reference.

If using direct EPUB upload, the API user will POST a file’s contents as multipart/form-data using the parameter epub_data for the EPUB bytes to /api/documents/ (remember your api_key too). Here’s an example of this using curl:

#!/bin/bash

# Example Bookworm API client implementation to upload a file test.epub
# using curl.  This script expects two arguments: the Bookworm
# installation URL and a valid API key.
#
# curl must be installed (surprise).
# The Bookworm URL should have no trailing slash (e.g.
# https://bookworm.oreilly.com)

BW=$1
KEY=$2

# Upload a document
curl -iF epub_data=@test.epub -F api_key=$KEY $1/api/documents/ -D result.txt

If you’d prefer to add a new ePub by URL reference, POST a request to /api/documents/ with an epub_url parameter that references an externally-hosted ePub document available by unauthenticated HTTP GET. The Bookworm server will initiate a request for this ePub document and proceed as normal (maybe).

If either upload technique described above succeeds and the new ePub content is accepted by the Bookworm server, an HTTP 201 Created response (probably following some HTTP 100 Continues) will be returned and the Content-Location header will reference the document address (write that down). The newly-uploaded title will now be available in the user’s Library .

Neither method for adding new ePubs checks to see if the user already seems to have this title in their Library. API clients should check the contents of the user’s Library listing before adding new titles.

Downloading ePub content from your Library

API users will be able to download complete ePub documents from the user’s Library by document address. An authorized HTTP GET of a document address will return the complete content as application/epub+zip.

Assuming you have a Flash movie called Creative_Commons_-_Get_Creative.swf and a containing XHTML page called chapter-1.html, put this in your OPF:

    <!-- Flash video -->
    <item id="video"
          href="chapter-1.html"
          media-type="application/xhtml+xml"/>

    <item id="video-flash"
          href="Creative_Commons_-_Get_Creative.swf"
          fallback="video-fallback"
          media-type="application/x-shockwave-flash"/>

The first item is a standard OPS document; the second is the Flash movie itself. Notice that we declared a fallback to another item. That item needs to be a supported type, like an image:

    <item id="video-fallback"
          href="getcreative.gif"
          media-type="image/gif"/>

Then we need to create the XHTML document, including the Flash movie. The document should be valid XHTML 1.1, which precludes using many of the boilerplate Flash code available on the web which is meant to support IE or other legacy browsers. This code is valid and works in Bookworm (under Firefox):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Flash video test</title>
  </head>
  <body>
    <p>
      <object type="application/x-shockwave-flash" data="Creative_Commons_-_Get_Creative.swf" width="300" height="400">
        <param name="movie" value="Creative_Commons_-_Get_Creative.swf" />
        <img src="getcreative.gif" alt="banner" />
      </object>
    </p>
  </body>
</html>

Notice the implicit fallback img as well.

Try out the video demo, which works in Bookworm and Adobe Digital Editions. The ePub is released under a Creative Commons Attribution license.

Edited November 15, 2009 to provide a working version in Adobe Digital Editions; thanks Dave Cramer.

Whether the video will actually play depends on the epub file’s formatting and the capability of your web browser. Bookworm also needs to register the particular file type.

Supported video formats:

• MP4
• OGV
• SWF

In later posts I’ll demonstrate how to create a valid ePub containing embedded video in a variety of formats.

Please report issues with video embedding to the Bookworm issues list. You must include a sample epub for testing.

Most of these are (by design) internal and not user-facing, but at least two are notable and possibly quite fun.

Feedbooks integration

At the bottom of the user library page (your home page when you log in), there’s now an option to download any books from the Feedbooks most-popular feed. This is language-aware; Bookworm will request books in your current language preference, and Feedbooks will return the most popular books in that language:

One-click adding

Clicking on a Feedbooks title will immediately add that book to your library, without having to download it to your computer first.

Any other site can create an “add to Bookworm” button using this simple URL scheme:

http://bookworm.oreilly.com/add/?epub=http://www.example.com/book.epub

For example:

http://bookworm.oreilly.com/add/?epub=http://www.feedbooks.com/book/187.epub

At this point, you will already need to be logged in to Bookworm for this to work. In the near future we’ll provide even more methods to get content in and out of Bookworm.

Many thanks to the O’Reilly TOC folks who sponsored this work.

Thank you Mahalo Ito!

I’m thrilled to finally have this up as Spanish was one of the languages I was most interested in adding.