I’ve added some command-line options to override the location of the DocBook XSL (previously it was stuck in a settings file) and to also point to an optional CSS file.

$ db2epub.py
Usage: db2epub.py docbook1.xml [docbook2.xml]... --xsl [DocBook XSL or customization] --css [css file]

Early in Bookworm’s history we didn’t specifically extract the identifier from the ePub and put it in the database, but we do now, so I can query the state of identifiers in the more recent books.

18,571 epubs in Bookworm have some kind of identifier in the database. Many of these were auto-generated by Bookworm, for example if the book had none at all.

14,735 of these are unique. Most of the duplicates came from earlier versions of Calibre, which simply generated identifiers like “25.” Others are just duplicate copies of the same book, which of course is okay.

There are three truly useful identifier types. I’ll specify these in order of my personal preference:

ISBN

Modern commercial books already have unique identifiers. Convenient! Use the ISBN as your primary ID in the epub if you have it, and be sure to use the correct ISBN for your ebook edition:

<dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/"
                 id="bookid"
                 opf:scheme="ISBN">urn:isbn:9780596158347</dc:identifier>

Including the scheme “isbn” in the identifier as well as using the optional opf:scheme attribute allows intelligent reading systems to leverage the ISBN by searching other systems. (1,132 books in Bookworm are courteous in this way.)

URI

Project Gutenberg epubs use a URI. This is a great method for digital-native books with their own steady identifiers. In this case Gutenberg identifiers include the Gutenberg book id:

    <dc:identifier opf:scheme="URI" id="etextno">http://www.gutenberg.org/ebooks/11</dc:identifier>

Like ISBN, these identifiers are stable, and let reading systems do more with the book if they’re able (for example direct users to the canonical information page for that book). If all publishers had their own websites with a page for every book I’d even prefer URIs to ISBNs for this reason.

939 books on Bookworm are from Gutenberg directly, based on searching for this style of identifier. Only about 100 other books use URIs as identifiers.

UUID

A Universally Unique Identifier solves the “required unique” problem by specifying very large numbers, using a variety of schemes.

It’s trivial to generate a UUID in most programming languages. Here’s Python:

>>> import uuid
>>> uuid.uuid4()
UUID('58dce2ac-7aec-45c3-a6de-903a30061545')

Wikipedia lists UUID implementations in other programming languages. You can even just go to a UUID-generating web site. These days, if Bookworm gets an epub with no identifier, it generates a UUID.

UUID provides the least useful information of the three identifier types, but, as Wikipedia points out, “One’s annual risk of being hit by a meteorite is estimated to be one chance in 17 billion [...], equivalent to the odds of creating a few tens of trillions of UUIDs in a year and having one duplicate.” So at least it’s unique.

Submit a patch

Of course the holy grail for the lazy open source developer is when someone submits a “patch”, or a fix for the issue. That means you don’t have to do any work because someone fixed it for you. Hooray!

In practice, this is pretty rare except for the most active open source projects. Many of the users of these tools aren’t programmers, or even if they are, they don’t have the time to go digging through unfamiliar source code to find the source of the problem.

Post a bug, don’t send an email

All open source projects have public bug trackers. This is probably the best thing about open versus closed source. Do you know where to submit a bug to Adobe Digital Editions? I don’t. But I know where to send an epubcheck bug.

It’s common courtesy to quickly scan the bug list to see if your issue has already been reported, but everybody understands that sometimes this is inconvenient. The Calibre bug tracker has more than 500 open tickets. Nobody expects you to go through them all, and it’s easy for the developer to close duplicates. But if it’s a small list like epubcheck’s, do check first.

Reserve sending emails to the developers for when you actually want to open a dialogue, or need to discuss something privately.

Submit a test case

When you post a bug, the best way to ensure your bug gets attention is to submit a test case. In the case of ebooks this is pretty easy — submit a sample ebook that exhibits the problem.

A good test case should be as minimal as possible. Ideally it should have only the content necessary to exercise the bug.

Generating a test case can be time-consuming, but remember you’re asking for someone else (usually a volunteer) to take anywhere from 15 minutes to several hours to fix your issue.

Consider the human

The key concept here is that submitting a bug is a social interaction, not a technical one. You want to convince a developer who is probably busy and distracted that your bug is worth their time.

Open source bugs are often fixed in the short intervals that people have between other tasks. You want to convince the developer that the bug can be fixed in that timeframe. Providing a clear bug report, being polite, and providing a test case are the best ways to support that assertion, even if in the end it turns out to be a tough problem that takes hours to fix. Hook them with a clear test case!

As in the earlier post about bash ePub tools, let’s create a shortcut:

function epubcheck()
{
curl --silent --data-binary @$@ http://threepress.org/epubcheck-service/
}

Parsing the validator output

The validator will just return raw XML to the console; if you wanted to produce more human-readable output, use this XSLT:

epubcheck.xsl

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
      exclude-result-prefixes="xs"
      version="1.0">
  <xsl:output method="text" media-type="text/plain" encoding="UTF-8"></xsl:output>
  <xsl:template match="/">
    <xsl:choose>
      <xsl:when test="//is-valid='False'">
        <xsl:text>File was NOT valid!</xsl:text>
        <xsl:text>&#xa;</xsl:text>
        <xsl:for-each select="//error">
          <xsl:value-of select="."/><xsl:text>&#xa;</xsl:text>
        </xsl:for-each>
      </xsl:when>
      <xsl:otherwise>
        <xsl:text>File was valid!</xsl:text>
        <xsl:text>&#xa;</xsl:text>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>
</xsl:stylesheet>

Save that as epubcheck.xsl then refactor the function to capture the output from the API and run it through the XSLT:

function epubcheck()
{
curl --silent --data-binary @$@ http://threepress.org/epubcheck-service/ > /tmp/epubcheck;
xsltproc epubcheck.xsl /tmp/epubcheck;
rm /tmp/epubcheck
}

Now you’ll get an interaction that’s easy to read:

$ epubcheck code.epub
File was valid!
$ epubcheck malformed.epub
File was NOT valid!
ERROR: temp.epub: OPF file OEBPS/content.opf is missing
Check finished with warnings or errors!

But try to get epubcheck running locally first, as you’ll get better performance and save me some bandwidth.

Install these functions in your .bash_profile. This should work on any Unix-like system including OS X.

Zip up an ePub file

function epub()
{
rm -f $@; zip -q0X $@ mimetype; zip -qXr9D $@ *
}

Call this as:

$ epub my-book.epub

from the directory where the resources are (meaning, the should be at least a mimetype and META-INF in the current working directory). The actual arguments were taken from Bob DuCharme’s great post on creating .epub files with simple tools.

Validate an ePub file

function epubcheck()
{
java -jar /path/to/epubcheck.jar $@
}

$ epubcheck my-book.epub
Epubcheck Version 1.0.3

No errors or warnings detected

$ epubcheck my-book.epub
Epubcheck Version 1.0.3

ERROR: malformed.epub: OPF file OEBPS/content.opf is missing

Check finished with warnings or errors!

And then I like a shortcut that does both at once:

function ev()
{
epub $@; epubcheck $@
}

Tomorrow I’ll show how to do the same thing even if you encounter errors running epubcheck locally, using the newly-available ePub validation API.

Our web tool for using the open-source epubcheck validator got a quick refresh:

Now includes epubpreflight results

EpubCheck’s lesser-known companion checks for additional issues like content documents that exceed 300K, which can’t be loaded on the Sony Reader. The complete (short!) list:

• Content files that are empty or are >300k
• Image files that are empty or are >10MB

Simple API service

This has been part of Bookworm for some time but I’m finally documenting it: there’s also a machine-readable API for the validator. It’s very simple — you get back an XML document indicating whether the file was valid, and if invalid, the list of errors.

Just POST an epub file as a series of bytes to http://threepress.org/epubcheck-service/ and look at the response. Here’s a sample interaction using curl, but you could use your favorite HTTP client.

Successful validation

$ curl --data-binary @pg345.epub http://threepress.org/epubcheck-service/

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
   <is-valid>True</is-valid>
</rsp>

Invalid ePub

$ curl --data-binary @invalid.epub http://threepress.org/epubcheck-service/

<?xml version="1.0" encoding="utf-8" ?>
<rsp stat="ok">
    <is-valid>False</is-valid>
    <errors>
        <error>ERROR: temp.epub/metadata.opf(6): attribute "files-as" from namespace "http://www.idpf.org/2007/opf" not allowed at this point; ignored</error>
        <error>ERROR: temp.epub/metadata.opf(9): unknown element "series_index" from namespace "http://www.idpf.org/2007/opf"</error>
     </errors>
</rsp>

In the last few years many technology blogs have been participating in “National Blog Posting Month,” a play off the more famous National Novel Writing Month.

Threepress is participating: we’ll be posting ebook technology tips, code samples and updates on our projects every day for the whole month of November. Hopefully we’ll include some guest bloggers as well as some posts from individuals who’ve worked on Threepress projects.

I’ll start off with five tips for ebook conversion.

Try the Calibre command-line tools

If you’re a developer or just impatient you’ll probably prefer to use the command-line tools included in Calibre rather than the main GUI application, which can be slow to start up and quick to crash. Many people don’t realize they’re available, but they’re great for one-off conversions or embedding in other applications. The most useful tool is the one-stop shop ebook-convert (older versions of Calibre had individual scripts for different formats).

The complete list and documentation for all the Calibre command-line tools is available from the Calibre site.

Prefer InDesign to Word, Word to PDF

If you have an InDesign original and at least version CS4, definitely start by creating an ePub from it. You’ll still need to do some hand-work in most cases, but it will save a lot of effort.

Word has its problems, but it’s more easily converted to a reflowable format than PDF. If you have only those two, start with the Word file. I’d probably do Word->RTF->ePub, with the last step in Calibre.

Convert from more complex formats to less complex ones

Mobipocket has simpler markup than ePub. We recommend that if you need to produce both formats you make the Mobipocket from the ePub file, rather than the other way around. XHTML in ePub is a more rich expression language than what’s available in Mobi, and is less likely to lose formatting details.

Test in multiple devices and form-factors

When we test ePub books we’ve created, this is our go-to list of readers:

1. Bookworm
2. Adobe Digital Editions
3. Smartphone reader (Stanza on the iPhone)
4. e-ink device (Sony Reader)

We usually start with Bookworm running under Firefox; you’ll get the best CSS debugging that way, especially when using Firebug.

Validate

Most people working with the ePub format are aware of the EpubCheck validator created by engineers at Adobe. (The web-hosted epub validator on the Threepress site is by far our largest driver of traffic.) But I mention it again because it’s important! Peter Sorotokin from Adobe has been committing changes recently, so this is a good time to get any requests or bugs into the issue tracker.

Later this month we’ll have some interesting updates to the Threepress version of the validator that should make checking easier and more thorough.

(Edited Nov 1 to reflect updated information about Calibre tools)

Users can comment using the web site, read the entire in-progress text of the book (updated as the authors revise it), or follow comments via an Atom feed. Readers who comment will be credited in the final edition of the book.

Authors do not ever need to administer or even visit the web site, as they receive the comments incorporated back into the manuscript source. To my knowledge this is the first public commenting system that works in this way, using
DocBook XML to both render the content and integrate the feedback in a contextually-relevant manner. It wouldn’t be possible without XML.

OFPS was built using the DocBook XSLT, Subversion version control and the Pylons web framework. Two books were on my lap constantly: The Definitive Guide to Pylons (also available for free online!) and Essential SQLAlchemy (I was a technical editor on this but apparently forgot it all).

More information in the post announcing this on the O’Reilly Labs blog.

Paul Norton from Adobe has contributed a new software suite to the epub-tools project hosted on Google Code. The Java library provides tools to convert from an impressive number of formats: Word, RTF, and FictionBook.

Here’s Paul’s description of the project:

… the main intent of this project is to provide source code and examples of the way things could be done. In other words, there’s plenty of room for developers to improve and enhance the conversion.

The project includes code to convert a couple different formats to ePub, including generating all the required files and creating the package. The project also shows how to mangle embedded fonts, how to sub-set those font (thus reducing the size of the ePub).

More on the Digital Editions blog.

I think it’s great that Adobe is supporting ePub in this way, both with additional developer tools and the invaluable epubcheck. Thanks to Paul and to the Digital Editions team!