The result is Caveman, a Python tool to validate HTML5 cache manifests. It scrapes the HTML page you specify, finding resources, then compares them to the cache manifest and reports problems.

Full blog post and complete documentation on Caveman.

Part two: Scrivener

I first became aware of Scrivener’s support for EPUB export in November of 2010, when they released a special edition for NaNoWriMo (and the less said about my novel-writing experiment, the better). EPUB export is now available in the latest major release.

Though Scrivener is primarily a Mac application, there are beta versions for Windows and Linux. I have no idea if they support EPUB export — please comment if you know either way!

The sample document

Scrivener’s interface can be intimidating if you have only worked with relatively straightforward word processors. It’s less of a document editor than a book authoring platform (or the way I would look at it, it’s Visual Studio, not emacs).

For this test I used the “Manuscript” style, which provides many of the utilities that an author of a novel-length book might want, rather than starting with an entirely blank document (analogous to using the EPUB sample document in Pages). For one thing, the Manuscript layout already provides obvious support for structured blocks like chapters and parts. In fact Scrivener segments documents even further into “scenes” which can be relocated easily through the document. Programmers love structured content!

As in the previous post, I set up the sample document to include all of the obvious stylistic elements that I wanted to test. I didn’t attempt to exactly re-create the Pages sample as that would be soul-crushing.

Finding the EPUB export feature can be challenging; it’s actually listed as “Compile.”

Export options

No complaints here. There are dozens of options for EPUB configuration from the menus.

EPUB-specific output options:

Select which content segments are included in the book, and in what order:

My favorite pane. Look at all that metadata!

Output

Scrivener’s output is variable in quality. Some things are nice. It’s valid, for one thing, and I was able to produce this valid output without any special handling — I didn’t change any of the menu items beyond adding the metadata, which indeed trickles down nicely into the OPF file:

       <dc:title>Creating an EPUB sample document</dc:title>
        <dc:identifier id="PrimaryID">urn:uuid:A54DCD13-A6E0-4ACA-8B31-4E0EBA0624EF</dc:identifier>
        <dc:language>en</dc:language>
        <dc:creator opf:role="aut">Liza Daly</dc:creator>
        <dc:subject>Non-fiction</dc:subject>
        <dc:description>An EPUB test of Scrivener's output</dc:description>
        <dc:publisher>Threepress Consulting Inc.</dc:publisher>
        <dc:rights>http://creativecommons.org/licenses/by-sa/2.0/deed.en</dc:rights>
        <dc:date>2011-05-29</dc:date>

Whitespace is exported as significant. While this makes the web developer in me cringe (I’d rather see CSS margins used here), it’s acceptable, especially since whitespace is semantically relevant in fiction, as when used for scene breaks:

<p class="p1"><br /></p>
<p class="p1"><br /></p>
<p class="p2">Hello world!</p>
<p class="p2">I have lists and tables!</p>

Unlike Pages, lists are generated as the correct list elements:

<ol class="ol1">
  <li class="li3">Numbered lists are numerous.</li>
  <li class="li3">We often have more than one item.</li>
</ol>
<ul class="ul1">
  <li class="li3">Make love</li>
  <li class="li3">Not bullets</li>
</ul>

Tables are fine, though I’d like for these tools to have a “table header” style that could ultimately map to th elements. This is a little more readable than the Pages output.

<table cellspacing="0" cellpadding="0" class="t1">
  <tbody>
    <tr>
      <td valign="top" class="td1">
        <p class="p2">Not sure how to make headers</p>
      </td>
      <td valign="top" class="td1">
        <p class="p2">Without just using colors</p>
      </td>
      <td valign="top" class="td1">
        <p class="p2">Not semantic</p>
      </td>
    </tr>

Headers are curiously not optimal:

<h2 style="margin: 0.0px 0.0px 0.0px 0.0px; font: 138% Optima"><b>First, a header.</b></h2>
<h3 style="margin: 0.0px 0.0px 0.0px 0.0px; font: 92% Optima"><b>A subheader.</b></h3>

Why the inline margins and font? Why the b? And like Pages, no apparent way to generate strong and em:

<p class="p2">Back to body text. But with <b>strong</b> and <i>emphasis</i>?</p>

Drag-and-drop images and internal and external hyperlinking all worked well:

<p class="p2"><span class="s1"><img src="images/droppedImage.png" width="272" alt="Image" /></span><span class="s2">Hello dog.</span></p>
<p class="p4">Inline URL: <a href="http://www.flickr.com/photos/beinecke_library/5166412915/in/set-72157625240109163/">http://www.flickr.com/photos/beinecke_library/5166412915/in/set-72157625240109163/</a></p>
<p class="p4">A hyperlink to <a href="body1.xhtml">Chapter One.</a> A hyperlink to <a href="http://placekitten.com">a website</a>.</p>

I pulled the image out of the Pages EPUB output – Pages renamed it to droppedImage.png, and Scrivener retained the filename, which is a nice touch.

Room for improvement

• Scrivener seems to generate one CSS file for each XHTML file too many CSS files by default. Uncool! This could potentially generate huge unwieldy EPUB books. There’s no reason to have more than one CSS file in an automatically-converted document, and the styles should be normalized across XHTML files.
• Like Pages, there was no obvious way to add alt text values to images.
• Also like Pages, I’d like to be able to customize the elements and class names in the outputted XHTML using simple menus.

A pleasant surprise to me was that Scrivener has a lot of options for power-users under the hood, including integration with version control tools like Subversion, use of a text-based markup format for better serialization to other formats, or low-level customization of the CSS or even XSLT: Scrivener advanced topics. This means it’s potentially a better choice for an ambitious digital-only publisher; you can start with the basic WYSIWYG layer, and gradually customize the output by diving into the lower level. This will be especially true once Scrivener is fully cross-platform.

Recommended with reservations (but mainly because of the CSS file issue).

Download the sample EPUB file and Scrivener file (unzip first). (Released under a Creative Commons Attribution license)

The IDPF board met on the last day of the Digital Book 2011 conference at Book Expo America. One of our topics for discussion was what the IDPF as an organization should do to further the adoption of EPUB. I brought up an issue that’s been concerning me for some time: the lack of digital-native authoring tools aimed at authors, not publishers.

If publishers are struggling to produce high-quality EPUB files either via InDesign, XML workflows, or strategic outsourcing, authors are in an even worse place. This is especially true for authors with an ambition to self-publish, or to start a micro-publishing outfit, and yet still retain some creative control over the look of their digital product. InDesign (especially CS5.5) is a great solution for small- to medium-sized publishers who produce both print and digital books, but its feature set is inappropriate for digital-native publishing, and its price and complexity are unsuited for self-publishers.

I’m aware of two document creation tools right now that have native EPUB support and are available for my platform, Mac OS X: Pages, and Scrivener. (The only product I know of on the Windows side is Atlantis. Linux users have to make do with plugins for OpenOffice — judging from the comments in the issue tracker, EPUB export is not a priority, to say the least.)

This post will cover Apple Pages. A subsequent post covers outputting EPUB with Scrivener.

Pages

Apple’s Pages was the first major commercial word processor to include EPUB export. I reviewed the initial EPUB support in August 2010, but it’s been through some updates since then, and I wanted to dive into the semantics of the outputted code more closely.

The sample document

I started with the Apple-provided EPUB template (more on that later) and added a number of new elements and semantic tests. In particular, I added:

• Chapters and headings
• Emphasis and strong text (rendered in Pages as italics and bold)
• Numbered and unnumbered lists
• Hyperlinks both internal to the document and external to the web
• Inline images (by dragging and dropping)
• A cover page with an image
• All available metadata in the export pane
• Tables

In all cases, I used only styles available in the style drawer; I did not change any font sizes or font weight via the toolbar buttons.

The EPUB output

As in my previous test, this produced a valid EPUB 2.0.1 document according to EPUBCheck 1.1. Hooray!

Headers and subheaders

The semantics are much-improved from my first test. Paragraphs are now wrapped in <p> elements, for example, and headers are headers:

  <body>
    <div class="body" style="white-space:pre-wrap">
      <h3>Chapter Two: The Chaptering</h3>
      <p class="s2">This chapter has an introduction. Hello!</p>
      <h4>I’m a subchapter or section under that. </h4>
      <p class="s2">Don’t hold it against me. I just have a lot of things to say.</p>
    </div>
  </body>

However, the white-space:pre-wrap style is curious: the property is meant to specify that whitespace inside the XHTML is significant, meaning that the ereader/web browser should retain it. That is emphatically not a best practice in general text; on the other hand, there was no whitespace in the output at all, so I’m unsure of its purpose. If I were post-processing this EPUB file, I would remove that style.

I used the “Chapter” style to generate the chapter heading. This header should be an h1 rather than an h3, but at least the subheading is also a header and one step down.

Go boldly

I completely failed to find a way to output strong and em rather than b or i.

Listless

I used the list styles provided in the template, but these are not the lists you’re looking for:

      <h3>Chapter Three: Lists</h3>
      <p class="s2">Reasons why people love lists, in order.</p>
      <p class="s2 s3"><span class="c2">1.</span>Lists are neat.</p>
      <p class="s2 s3"><span class="c2">2.</span>It’s cool to let the computer fill in numbering.</p>
      <p class="s2 s3"><span class="c2">3.</span>Yessir.</p>
      <p class="s2">Other reasons that people like lists, in no particular order:</p>
      <p class="s2 s4"><span class="c3">•</span>Sometimes they have bullets</p>
      <p class="s2 s4"><span class="c3">•</span>Not real bullets.</p>
      <p class="s2 s4"><span class="c3">•</span>Those are scary.</p>

This must be fixed.

Tables

A little verbose markup-wise, but basically fine:

      <table class="s5" style="margin-left:0.0px;width:99.8%;border-collapse:collapse">
        <col style="width:33.3%"/>
        <col style="width:33.3%"/>
        <col style="width:33.3%"/>
        <tr style="height:25.0%">
          <td class="s8 s6 s7">
            <h2 class="s9">Reasons why tables are nice</h2>
          </td>
          <td class="s8 s6 s7">
            <h2 class="s9">Who feels this way</h2>
          </td>
          <td class="s8 s6 s7">
            <h2 class="s9">I can’t think of a third thing.</h2>
          </td>
        </tr>
        ....

Images, covers, and links

Creating an image is as easy as dragging it in. I’m not sure if it’s possible to add alt text to the image — I believe document creation tools should prompt users to add descriptive text by default.

      <p class="s2">
        <img src="images/droppedImage.png" alt="droppedImage.png" style=""/>
      </p>

Only images styled as “inline” will be exported; Pages will warn you that the image was discarded if it had a floating or fixed style. I tried to select a page with an inline image as the cover page but Pages gave me a warning that it was being discarded. Then it actually shows up in iBooks anyway.

It would be nice if the original filename were preserved (it was not “droppedImage.png”), and the empty style attribute should be discarded on output.

      <h1><span id="chapter-5-sh1"/>Chapter Five: Hyperlinks</h1>
      <p class="s2">This is an internal link to <a href="chapter-1.xhtml#b1"><span class="c1">chapter one</span></a>. This is an external link to <a href="http://placekitten.com/"><span class="c1">photos of kittens</span></a>.</p>

The empty span here is for the purpose of creating a back-link. A similar one was auto-added to Chapter One. Adding an internal hyperlink requires an initial step of creating a Pages “bookmark”, and then linking to that bookmark, which was a little confusing; I should be able to target any point in the document using the hyperlink feature.

I didn’t test HTML5 video output, but I’ve been told that video can be successfully embedded and output such that the video will work in iBooks (it will use HTML5 tagging).

Metadata

Both the OPF and the NCX were perfectly well-formed. The EPUB export dialog should optionally request richer metadata than the current list of author/title/subject, though.

The dreaded sample document

The EPUB export function is next to useless on large documents unless you start with the sample template, or import its styles later and tediously update yours to match. The EPUB styles are completely opaque — I have no idea why they have magical properties, or what I would do to my own styles to emulate them. Since the Pages native file format is binary, there’s nothing for me to inspect to reverse-engineer the styling. The Pages file format is zipped XML, so it may be possible to inspect it directly — thanks Steve!

The native header/paragraph/list styles in the blank template should output useful semantics in the XHTML. It is unacceptable to force users to import an external document to produce a half-decent EPUB file. At the very least, an EPUB-friendly template should be one of the default choices available when creating a new document.

Improvements

• The list styles should generate lists. They should be ordered or unordered as appropriate to the style.
• EPUBs should be importable as well as exportable. It’s understandable that they won’t magically re-constitute into the original Pages document, but a conversion pipeline is entirely reasonable.
• It should be possible to export chunked EPUBS (with multiple XHTML chapters) without having to use the sample template.
• It should be possible for a power user to understand how to create styles that will have specific behaviors.
• It should be possible to customize the XHTML serialization (“I want the style named ’strong’ to output strong elements with the classname ‘foo’”).
• There should be much more metadata allowable in the OPF file.
• Images should require or at least prompt for alt attribute values.
• Bold and italic buttons should output strong and em with the appropriate CSS styling in all cases. I would say this is actually true of any EPUB output tools — it’s unreasonable to ask users to create named styles (as in InDesign) when those tempting bold and italic buttons are available.

I don’t expect Windows/Linux versions of Pages to ever exist, which means that Pages will remain a marginal tool in the publishing ecosystem, but it’s perfectly adequate for an individual Mac-only user.

Download the sample EPUB file and sample Pages document, (released under a Creative Commons Attribution license).

The original template now produces <p> elements instead of <div>s, an important update:

<h1 class="s7">Introduction to ePub</h1>

<p class="s6">This document will show you how to use paragraph styles to create a publication that looks great and is easy to navigate in ePub readers like iBooks... </p>

Presumably this affects the degree to which Pages-outputted styles are visible in iBooks (as user preferences may override), but it’s great to see more semantically-correct output. This corrects my earlier reservation about using Pages for EPUB creation.

First up is internationalization. One of the key advantages of an HTML5 webapp versus a native app is that they can be installed in any country on a supported device, even in countries that don’t yet have an Apple App Store. But first the application needs to be localized for the regional language.

Ibis uses the Django application framework and Django comes with native internationalization support. We also used this when internationalizing Bookworm with the help of some great volunteers.

Before translators can get to work, the application needs to be readied: each bit of text in the site needs to be marked for translation. This step can be time-consuming and difficult to test.

Automating translation testing

Ned has posted some code on his own blog that he used to help validate the internationalization process. Some good comments point to more advanced tools that take a similar approach. Recommended reading for anyone doing internationalization work.

The tool fakes a translation by randomly capitalizing all of the letters in the site text. If done properly, you should get a result like this:

The book title Middlemarch and the chapter name look normal; those should not be translated. The remaining words are directions to the user and will need to be in a regional language. If any of the site text, like “Next” or “Previous”, showed up with normal capitalization, we would know that we missed a step.

We won’t be rolling out translated versions to ibisreader.com in the near future, but we will be posting some performance enhancements and feature updates very soon.

If you’re making more than 3 EPUBs this year, you must setup epubcheck on your computer

Why setup epubcheck locally? If you have a local copy, you can:

• run it even if the online validator is down (it was this morning)
• run any version of epubcheck you want (useful when new versions are near release)
• run it more quickly
• run it on a lot of documents quickly (with a little command-line-foo)

Warning

This is a brief description of how you might be able to run epubcheck locally. This isn’t a tech support site, so please don’t post system questions in the comments.

Setup for both platforms

1. Ensure java is installed and in your PATH (Google is your friend if not)
2. Download the “binary” release you’d like to use (probably the featured one) from the epubcheck downloads page
3. Unzip the downloaded .zip file to somewhere on your computer
4. Find the file epubcheck-[something].jar in the newly unzipped folder (and keep this window open)
5. Find the EPUB file you’d like to validate (and keep this window open)

Each time you’d like to run epubcheck, you’ll need to ask the java command to run the epubcheck program (packaged as a .jar file) against your EPUB file. Happily, you can get drag and drop to help you with most of the filenames.

Windows

1. Open the “Run…” menu using [Windows-key]+r, Start Menu->Run…, or turning it on for Vista.
2. In the “Run…” menu, type cmd.exe.
3. In the new Command Prompt window (the “prompt”), type java -jar (note trailing space after “-jar”).
4. Drag the epubcheck .jar file from the unzipped folder into the Command Prompt window (the location of the file ["path"] should be copied).
5. Type another space in the terminal window after the .jar filename.
6. Similarly, drag the EPUB file from wherever into the Command Prompt window (it’s path should also be copied).
7. Press Enter.
8. Note the results in the Command Prompt window (No errors or warnings detected means it passed).

Mac

1. In the Finder, to Go->Utilities, then find the Terminal application (or do a Spotlight search for “Terminal”) and start it.
2. In the new Terminal window (the “prompt”), type java -jar (note trailing space after “-jar”).
3. Drag the epubcheck .jar file from the unzipped folder in the Finder into the Terminal window (the location of the file ["path"] should be copied).
4. Type another space in the terminal window after the .jar filename.
5. Similarly, drag the EPUB file from wherever in the Finder into the Terminal window (it’s path should also be copied).
6. Press Return.
7. Note the results in the Terminal window (No errors or warnings detected means it passed).

Linux

Ha, you’re on your own! (But do see the good comment from Toby Stevenson, below.)

Other tools

Now that you’ve mastered the above, you can try to run the epubpreflight checking tool in the same manner. It helps check if files are too big for the older Sony Reader devices, for example.

Note: There is a new jar file in lib/. Ensure you copy that to the same place as your existing saxon.jar.

Threepress sponsored the following fixes:

1. Error message generated for valid ISO8601 date [BUG] Dates with timestamps were being incorrectly flagged as invalid.
2. Checking for unused files which are not in the manifest [ENHANCEMENT] EpubCheck will now issue a warning if files are in the package that are not in the manifest or META-INF.
3. Existence of files referenced in guide not checked [BUG]
4. Valildation problems with guide and fragment identifiers [BUG]

The unmanifested file warning in particular has been a critical issue for publishers working in the EPUB ecosystem, as some retailers require that only manifested files be included in the package.

The date issue has also been problematic for EPUB files generated out of tools like Calibre that produce dates with timestamps in them.

Our web-based validator will be updated to epubcheck 1.1 shortly.

Error message generated for valid ISO8601 date [BUG]
Checking for unused files which are not in the manifest [ENHANCEMENT]
Existence of files referenced in guide not checked [BUG]
Version r128 has broken test suite [BUG]

The unmanifested file warning in particular has been a critical issue for publishers working in the EPUB ecosystem, and we believe this update alone warrants a new public release and a version number push. The date issue has also been problematic for EPUB files generated out of tools like Calibre that produce dates with timestamps in them.

Download epubcheck 1.1-rc1: epubcheck-1.1-rc1.jar. Note that you will need to know how to run the command-line epubcheck for this, and is intended for developers only.

Please report all issues to the epubcheck issues list, not this blog.

However, the vast majority of cases where this occurs are unwanted accidents: publishers including backup files, unused images, or operating system cruft like “.DS_Store.” It’s surprisingly easy to generate very large .epub files due to multiple backups, so while it’s not a formal requirement of the spec, it’s a reasonable best practice.

We consulted with other EPUB developers and have contributed a patch to epubcheck to flag files that are missing from the manifest as a warning. There are three exceptions to this rule:

• Required files that aren’t meant to be in the manifest, like mimetype.
• Any files listed in META-INF/container.xml, including the OPF file itself.
• Any files found in the META-INF directory.

All other unmanifested files will generate a warning:

WARNING: ./general/Unmanifested.epub: item (Unmanifested.txt) exists in the zip file, but is not declared in the OPF file

More details available in the original issue.

We’d like the community to review the patch to ensure that we’ve caught all the necessary cases. If you find a problem, report it as a formal bug, not as a comment here (or do both!). Please include a sample EPUB file as an attachment in the issue so we can generate a test case to verify the fix.

You can download the latest code (revision 135+) either from Google Code as source, or just download epubcheck-1.0.6-dev.jar.

Because this is a developer preview, our online EPUB validator does not use this version. If you are not familiar with how to run epubcheck from the command-line, please continue to use the online version.

Do not use this build for production systems.

If no problems are found we propose that the formal revision number become epubcheck 1.1 to reflect the increase in functionality. We plan to continue work on additional outstanding epubcheck issues as well.

DocBook-XSL uses their experimental .0 releases, like 1.76.0, to encourage real-world testing. They should not be used in production systems, but are usually followed by stable releases, like 1.76.1, in a few weeks.

Some of the EPUB improvements include:

1. Support for multiple embedded fonts
2. Improved selection of cover images
3. Workarounds for ADE bugs
4. Changes to NCX documents to create better inputs for kindlegen (to create Mobis)
5. Better date outputs for epubcheck validation

Please report issues and bugs to the DocBook-XSL bug tracker.