Threepress Consulting blog

This morning my Nook updated to version 1.1.0. I ran through some of the tests in my original Nook/epub review. Other comments on the update are on Teleread.

Locations of My Documents change

I had loaded my local books in the Digital Editions folder, but after the update I had nothing listed on the Nook under My Documents. I moved them to the actual my documents folder and now they show up. I don’t know if this is a real problem for most people who probably put them in the right place in the first case, but it’s worth knowing if your books suddenly “disappeared.”

Rendering tests

Opening The People of the Abyss still takes a long time (about 30 seconds, enough that I initially thought it wasn’t going to work at all). Jumping into a chapter takes long enough that I get an OS-level error saying that the application is stalled (answering “wait” will eventually work).

• Going backwards into a previous chapter still shows no loading message, when loading can take time.
• Moving between chapters in this edition still takes 20-30 seconds.
• As described, pagination is now cached in the current book outside of the reading mode. Meaning, if you stop reading a particular book and browse the store, clicking Reading Now will return to the current book without re-paginating.

  However, going to a new book first and then returning to a previous book does trigger re-pagination, even if the settings have not changed.

• O’Reilly books still show a Formatting… message indefinitely, requiring a reboot.
• I didn’t see any improvements in foreign character set support. Books in Chinese still display properly in the metadata but “????” placeholders in the content.
• I do like that you don’t have to go through the “info” screen to read a book; selecting a title from the list goes directly into the content.

Possibly-new complaints

I can’t remember if it worked this way before, but when I’m in a book and I hit the back-arrow, I expect it to take me back to my list of books. Instead I get bounced to the main “flower” screen. This means there are multiple “clicks” required to move from one book to another and I found that very annoying.

I noticed that my library book (checked out for the previous report) has three days remaining according to Adobe Digital Editions on the desktop, but I can’t retrieve this information on the Nook itself. The book info page just provides other metadata like title and file path.

While I was reading one title (without complex CSS) it took multiple attempts to get the LCD screen to come back up by tapping on the screen. Using the hardware button did wake it up immediately. I hadn’t noticed this particular problem before.

So the changes as described in the Teleread post are nice, but the issues I had haven’t been addressed, unfortunately.

(There’s an updated post on the 1.1.0 firmware.)

This isn’t a full review of the device. In particular, I don’t cover purchasing books or reading PDB or PDF books at all. I was mainly interested in evaluating the Nook as a general-purpose ePub reader.

Adobe Mobile SDK

Like other e-ink devices, the Nook uses the Adobe Mobile SDK as its rendering system. This means that some of the same features and quirks found in Adobe Digital Editions and earlier e-ink devices like the Sony Reader should apply here.

However, I didn’t know whether the version of the Mobile SDK on the Nook would be any more recent or evolved than the ADE or Sony versions (I have a PRS-505). So I experimented with the ePub rendering in the way I would any new device.

Test set

As you might imagine I have a lot of ePub books lying around. I tried a variety of commercial and non-commercial books, some generated via InDesign or other automated processes, and some coded in XHTML/CSS by hand.

Library lending

The Boston Public Library subscribes to the OverDrive ebook offering and there are a number of titles in ePub format. I checked out a book, and when Digital Editions started up it discovered the Nook (connected via USB) and asked if it wanted to authorize it. I was then able to transfer the library book over with no issues.

It did take a minute to find the book on the device: books copied via ADE end up in My Documents rather than in the main B&N books list. Since it’s just a regular filesystem, I think many users will probably manually copy books into just one place to keep it simpler.

Top level filesystem on the Nook. Files uploaded via ADE end up in Digital Editions

Library book on the Nook. Hey, all the good books had holds on them.

Fonts

The Nook comes with three fonts: two serif and one sans-serif. In most ePubs I tried, changing the font face had no effect.

The image is blurry but you can vaguely see that while I have the font set to Helvetica Neue (which is sans-serif), the title and text are still in the serif font.

In another book I was able to change the font to sans-serif, but it didn’t apply equally throughout the content. A subset of the text that had its own styling remained serif:

Click to enlarge; see “The experiences” at the beginning of the chapter.

Font size changes did work as expected.

General UI comments

Changes in the local filesystem aren’t auto-detected. When you add a new book via USB, you have to manually “check for updates” to see the changes.

Selecting a book only brings up a rudimentary metadata page. While this is the kind of thing a nerd like me wants to see, does anyone else really care about the filesystem location of the book?

Only after selecting “Read” from the touch screen do you get to the cover image.

Language support

One of the things I was really hoping would be improved in the Nook relative to other Mobile SDK-derived products was international character support. Unfortunately, that is no better than ADE or similar readers:

Frustratingly, the Chinese book shown above did display Chinese characters in the listing of my library. I’m assuming this is because the Android OS is perfectly capable of reading the UTF-8 characters and has a Chinese font on-board, but the Mobile SDK does not.

I also tried an Arabic book with an embedded font, and while the font is displayed, the book’s text does not run right-to-left, and the necessary ligatures aren’t rendered. This is the same behavior as ADE. (Stanza and web-based readers like Bookworm display this same book correctly.)

Rendering performance

Here’s where things really fell down for me. I tried an edition of The People of the Abyss by Jack London. Mike Cane used this book to demonstrate some problems with ADE’s rendering.

This is a complex ebook though by no means an extreme outlier. It is hand-coded, which means that it doesn’t have unnecessary auto-generated markup, and it’s valid.

Many ereaders that paginate only render chapters on request, rather than at book loading time. That’s a good practice; it lets users get reading as soon as possible, and defers boring loading messages as long as possible. The Nook seems to be no different. When toggling between chapters (not pages), the user will generally get a “Formatting” alert; my guess is that it’s laying out each page based on the current font settings and content. You get a similar message when adjusting font settings, which also requires re-rendering.

Somewhat unprofessionally, the loading messages often aren’t middle-aligned correctly.

In most books I tested, the re-formatting between chapters or font changes took between 4-5 seconds. That’s slightly longer than the Kindle 1 or Sony PRS-505, which are the two devices I have for comparison.

For this particular book, re-formatting a chapter or going to a new chapter took 30 seconds.

Chapter renders are cached, but only in memory. If you close a book and re-open it, you get the same “Formatting” message again, and it will take just as long as before, even if you haven’t changed any settings.

(Jumping ahead into the book via the table of contents and then going “back” to a chapter you haven’t read before triggers the long rendering process, but no loading message is displayed. It looks like the device just hangs.)

This book demonstrated the same issues that ADE did in Mike Cane’s original post, so I don’t believe that the CSS support is any different in the Nook.

Loading failures

I tried loading a number of O’Reilly Media titles that are valid and work on the Sony Reader and every other ePub device. The Nook only brought up the “Formatting” message, and then hung. Only a full restart would bring it back.

This is an extremely serious problem.

Edited Dec 12, 2009 to reflect independent confirmation of this.

Hardware comments

I actually like the hardware quite a bit. While I’m right-handed, I like holding my ereader in my left hand and being able to turn pages that way. I prefer the Kindle 1 button placement to later Kindle versions, and I really dislike the hard, awkwardly-positioned buttons on the Sony 505. The Nook’s are easy to operate with either hand and don’t suffer from the extreme “bump” sensitivity of the Kindle 1.

I also didn’t have nearly as much of a problem with the LCD touch screen as other reviewers. Yes, the scrolling is quite slow and inaccurate, but single touch operations were responsive. Generally I found the interface intuitive, but your mileage may vary.

My one big complaint with the hardware is the weight. In the course of just 15 minutes of holding it, my hand got tired. I’m not sure the LCD screen is worth the strain.

Overall

I imagine that the books purchased on the B&N store won’t show these issues (especially those that are PDB rather than ePub with full CSS). But one of the key advantages of the Nook over the Kindle is its ability to read ePub from other sources.

While many of the rendering limitations are the same as those on the PRS-505 (now over a year old), I don’t have the same expectations in a new, more expensive device. The lack of support for non-English languages and non-Roman scripts is totally unacceptable, especially when the device’s operating system already comes with that support.

I do hope that the language issues and especially the performance bugs are resolved quickly, as non-B&N publishers may get hit hard on returns for non-functional books that are absolutely valid and should work.

Here are some frankly terrible iPhone camera photos I took as I struggled with unboxing the Nook, which appears to be shipped in some kind of impenetrable force field. Seriously, it took me 8 minutes just to open the thing between the first and last photo.

I’ll post a real report on the device after I’ve had time to evaluate it.

This is really funny if you get the joke, and still a nice style if you don’t. Thanks Kirk Biglione!

[Guest post by game designer and author Emily Short: see bio]

Authors and publishers alike are beginning to think more about the interactive possibilities of fiction.

The good news is that it’s not necessary to invent interactive storytelling from scratch. Computer games have been exploring this territory since the late 1970s; literary hypertext since the late 80s. These media are directed at a different audience from the book-buying public, but they do offer some pointers about what interactivity can add to a story:

Exploration. The reader chooses what to read about next, just as when browsing a website, following one of two possible reading orders for Julio Cortazar’s Hopscotch, or looking up references in Milorad Pavic’s The Dictionary of the Khazars. The order of reading doesn’t change the text, and a thorough reader may eventually see all of the work. Nonetheless, variable ordering makes the reading experience different for different participants.

Challenge. If the reader wants the story to end a certain way, but to achieve that, he has to figure out the right decisions to make. Challenge is a common component of stories in games. As a storytelling device, it can make the reader value certain outcomes more, or sympathize more deeply with the struggles of the characters. Presenting a challenge can also be an effective way to co-opt the reader: someone invested in trying to reach a certain outcome does not always pause to consider whether he should.

Complicity. When a story is interactive, the reader bears some of the responsibility for the way it turns out — and indeed for the fact that it comes out at all. Presenting the reader with options to continue the story, all of which are less than perfect, can force him to choose whether he wants to accept the universe as you’ve created it in your story or allow him to share — and understand more fully — the limits of a character.

Choice. The old Choose Your Own Adventure books let readers choose what the protagonist should do next: attack the dragon or run away? sell the jewel or give it to the beggar? But specific works in this model are often disappointing thanks not only to poor writing but also to badly-thought-out interaction.

There are many possible pitfalls. Options that separate into easily recognizable Good and Bad moral decisions tend to cheapen a story, turning it into a piece of propaganda rather than engaging fiction. On the other hand, so do choices where the reader has no reason in advance to care one way or the other — like asking him which of two identical doors the protagonist will pass through. Inconsistency is a problem too: choices in a story should generally be at a uniform level, about roughly the same kinds of decisions — not asking the reader to choose what to eat for lunch on one page and whose life to save on the next.

Done well, though, choices in stories can challenge the reader to explore his own beliefs.

Whatever angle you choose, you need to able to answer the question, “What does this interaction add to the story? What experience does it create for the reader that a non-interactive version could not?” What’s more, you need to give the reader enough information to make informed decisions, and enough grounding in the work to care about their outcome.

Further reading:

In general

“That Darn Conundrum” is an old but valuable post about three competing elements of interactive storytelling — freedom, agency, and well-formed story. Freedom, here, is the reader/player’s ability to do whatever he wants; agency, the degree of influence he has over the outcome of the story. These concepts are essential in analyzing and designing interactive narratives. The rest of the blog is good value as well. See also this post full of further references to work in interactive storytelling.

Marie-Laure Ryan’s Avatars of Story is an academic work and sometimes heavy going, but her chapter “Toward an Interactive Narratology” addresses many of the ways that fiction has been made interactive in the past. (My review of the book is here.)

“Homer in Silicon” is my biweekly column on interactive narrative in games. Of most general interest are probably the columns on interactivity misused in the HBO Imagine project; on interactive romance stories.

This list introduces a number of games and game-like works that I (or other commenters) thought were worth checking out for their interactive storytelling potential.

On choice

Chris Crawford on Interactive Storytelling is directed at authors of computer-based interactive stories, and some of Chris’ ideas are decidedly outside the mainstream of the computer game industry. Nonetheless, he opens with a very strong discussion of choices — well- and badly-formed — which is worth reading regardless of the medium of interactive storytelling you choose. (My review of the book is here.)

“The Mechanics of Morality” discusses video game approaches to moral choices, and some of the pitfalls that arise from these when they’re presented as black-and-white options. It’s very much a guide to what not to do.

Victor Gijsbers is a philosopher and game designer interested in the presentation of nuanced moral choice. He has a a website where he discusses his own text-based games about morality, “Fate” and “The Baron”. His blog is full of reviews and essays.

On exploration

The Reprover (from the French Le Reprobateur) is an interactive story that uses text, still drawings, and video to present a story from multiple angles. It is an excellent example of the power of exploration, because the open structure allows the reader to choose for himself what themes to pursue through the work.

Tale of Tales publishes odd, avant-garde video games that verge on not being games at all, and that reveal their meaning primarily through exploration and atmosphere. In my opinion, their most successful work is Fatale, a response to Oscar Wilde’s play Salome. (My review is here.)

Emily Short is an author of text-based games, conventionally called interactive fiction. She maintains a blog, writes about narrative-rich games for GameSetWatch and PlayThisThing, and is part of the team behind Inform 7, a language for creating interactive fiction. She can be reached at emshort AT mindspring DOT com.

I’ve examined all the listings in the DRM-free publishers index and:

1. Categorized the publishers, broadly, by genre
2. Indicated whether their ebooks are available as ePub

Both tasks were surprisingly hard: many of these publishers accept works from a variety of genres, and of course there are multiple distribution channels selling a variety of formats with or without DRM. My rule was to follow the most obvious links and hints about genre and formats.

I also removed a couple of publishers who probably did meet the DRM-free ebook criterion, but didn’t actually have prominent “buy” links or help information that could provide me with the information I needed.

Please continue to add listings on the DRM-free publishers page!

Obviously I’m a fan of the ePub format. It’s flexible enough to support advanced publications, but a simple text ebook can be put together with minimal effort.

But I don’t think it’s minimal enough. If I could go back in time and be involved with ePub and its predecessors, here are the choices I’d make:

Make the NCX optional

Many books or book-like publications have no chapters. In this case ebook authors are forced to create useless one-item NCX files and invent fake chapter titles, like “Pages.” Reading systems should be able to rely on the opf:spine alone to order OPS documents without defined, named chapters.

Simplify the NCX

Good progress is being made in the EPUB Working Group towards clarifying and simplifying the NCX requirements. (Making playOrder optional is an especially useful step.)

But because the NCX is derived from the DAISY specification, there’s still some useless overlap, like the duplicated title. NCX is great for complex hierarchies, but I wish its features were simply a part of the OPF file, leaving only one file with publication-specific metadata.

Drop container.xml and replace with a required name and location for the OPF

I don’t personally understand the choice that was made here: there’s a file with a required location and name (META-INF/container.xml) whose sole purpose is to point to another file which may be named arbitrarily. Perhaps this is a historical artifact, but why not simply require there to be a content.opf file at the top level of the publication and be done with it?

(When combined with my first change, this would result in ePub requiring half as many files, which to me is a good thing.)

Support any valid form of XHTML

XHTML 1.1 was obviously a mistake, as it’s used (as far as I know) nowhere else, and is a dead-end as far as web technologies go. Few automated HTML tools generate it, and the changes from XHTML 1.0 are simply annoying rather than useful.

I’d prefer that ePub support XHTML 1.0, which is simply HTML 4.01 with an XML vocabulary. HTML 4 is the dominant form of HTML on the web (and will remain so for some time) and common automated tools like Tidy can clean up “street” HTML 4 into XHTML 1.0 quite well. Tidy won’t, however, produce XHTML 1.1.

I also don’t want to put an upper bound on the XHTML supported: XHTML 5 should also be okay, and the rules for a reading system which don’t support later tags should be the same “ignore and move on” that has worked well on the web.

Minimize or eliminate any ePub-specific styles and markup

I dislike the existence of special style properties like the oeb-* styles where equivalent CSS3 properties exist. I know CSS3 is a mess, but I’d rather use the same vocabulary as will eventually be found on the web. (This was discussed but isn’t happening.)

Support MathML as a first-class document type

This will happen eventually, but who knows the timeline. At least the fallback system means that MathML documents are currently allowable; it’s just more of a hassle and many people don’t realize it’s possible.

How would you change the spec if you could magically make it so?

The ePub format allows for a fairly comprehensive set of book-friendly metadata, mostly drawn from the Dublin Core set of terms. Knowing what metadata to use and how it will appear in today’s and tomorrow’s readers is key; here are some recommendations:

Authorship

Two elements describe authorship of a work: dc:creator and dc:contributor. As you might guess, dc:creator designates the primary authorship, though there can be more than one.

Each author should be in their own element.. Don’t do this:

<dc:creator>James F. Smith, Thomas Jones</dc:creator>

Instead use:

<dc:creator>James F. Smith</dc:creator>
<dc:creator>Thomas Jones</dc:creator>

We recommend that anyone who had any kind of significant authorship role be marked as a dc:creator. Most reading systems use this element to discover or search for the author name, so ensure that individual authors in a multi-author work are so tagged.

A contributor designation is appropriate to describe individuals who had a non-primary role in the work. For example, if a multi-author work had multiple editors, or you want to capture additional staff like proofreaders, illustrators or the like, then use dc:contributor.

<dc:contributor>Alice Proofreader</dc:contributor>

File maker

ePub allows ebook creators to indicate how the values in these elements should be sorted. Name sorting, especially internationally, is a surprisingly difficult problem.

We recommend listing the author names in display order (usually First Middle Last) in the element text, and then setting opf:file-as to Last, First Middle, or whatever is appropriate in your source language.

<dc:creator opf:file-as="Smith, James F.">James F. Smith</dc:creator>
<dc:creator opf:file-as="Jones, Thomas">Thomas Jones</dc:creator>

Role playing

ePub extends Dublin Core by providing the opf:role attribute. This allows ebook creators to more precisely describe the contributions made by individuals listed in dc:creator and dc:contributor. It’s always a good idea to fill this out, even if not all reading systems display it:

<dc:creator opf:role="edt" opf:file-as="Smith, James F.">James F. Smith </dc:creator>
<dc:creator opf:role="edt" opf:file-as="Jones, Thomas">Thomas Jones</dc:creator>
<dc:creator opf:role="aut" opf:file-as="MacNeil, Lisa">Lisa MacNeil</dc:creator>

The recommended list of values for opf:role come from the MARC code list.

The vast majority of ebooks today have print cousins, despite some recent digital-only publishing news. As a consequence, many people creating ePubs want to know how to tie references to the printed pages back into the ebook. My personal opinion is that this sort of print-centrism is unnecessary for the vast majority of titles¹, but there are times when mapping the ebook to the printed book pages does make a lot of sense. Unfortunately, there’s no perfect solution at this time, but there are two options.

page-map

Adobe was motivated to provide a solution to this problem in Digital Editions before other reading systems, and they came up with a mechanism called page-map. The Adobe EPUB Best Practices Guide describes the issue:

There is no inherent linear navigation indicator which could be used for the same purpose that page number is used in the printed document world.

There is no way for an eBook to incorporate page number information for the printed edition of the same book.

Adobe developed an extension called page-map, documented in the sme Best Practices Guide, that provides a solution. To implement page-map, the creator includes a special page-map file in the ePub and references that file in the OPF metadata. This file gives a (page #) name to each pointer to a specific location within the content.

Here’s an example (note that many of the content files in the preface don’t span more than one page):

<page-map xmlns="http://www.idpf.org/2007/opf">
  <page name=""  href="strandedwithaspy_cov.html"/>
  <page name="" href="strandedwithaspy_intro.html"/>
  <page name="1" href="strandedwithaspy_fm01.html"/>
  <page name="2" href="strandedwithaspy_fm02.html"/>
  <page name="3" href="strandedwithaspy_tp01.html"/>
  <page name="5" href="strandedwithaspy_adc01.html"/>
  <page name="5" href="strandedwithaspy_ata01.html"/>
  <page name="6" href="strandedwithaspy_ded01.html"/>
  <page name="" href="strandedwithaspy_con01.html"/>
  <page name="7" href="strandedwithaspy_fm03.html"/>
  <page name="8" href="strandedwithaspy_fm03.html#page8"/>
  <page name="9" href="strandedwithaspy_fm03.html#page9"/>
  <page name="10" href="strandedwithaspy_ch01.html"/>
  <page name="11" href="strandedwithaspy_ch01.html#page11"/>
  <page name="12" href="strandedwithaspy_ch01.html#page12"/>

Pages in Action

How does page-map actually look in Digital Editions?

The catch? page-map is (intentionally) an extension to ePub and adding a page-map file to your ePub will make it invalid. On top of that, unless the reading systems is based on Adobe software (like Digital Editions and the Sony Reader), page-map will have no effect.

Pages in Digital Editions (without a page-map)

So, why do you always see these pages in Digital Editions, regardless? An un/fortunate feature of Digital Editions is the addition of the page-map-like display of pages, even if the ePub doesn’t include any page-map file. Here’s Adobe’s Best Practices again, describing how it chunks any content into a regular size, then labels each chunk a page:

When page map is not available in the document, Adobe Digital Editions will synthesize a page-map based on the document content. The approach used is the following:

Determine a compressed byte length of each resource which is referenced in the spine, subtracting any known encryption overhead (IV size)

Assume that there is a page for each 1024 bytes…

Some people want to see more of this, while others wish it could be turned off.

pageList

The NCX spec provides an alternative to the page-map extension, the pageList. This aptly named container for pagination information, provides a mechanism for giving a label (navLabel) to a point in the ePub (the pageTarget).

Here’s an example from an Internet Archive ePub:

  <pageList>
    <navLabel>
      <text>Pages</text>
    </navLabel>
    <pageTarget type="normal" id="pagetarget000006" value="6" playOrder="6">
      <navLabel>
        <text>6</text>
      </navLabel>
      <content src="part0000.html#page-6"/>
    </pageTarget>
    <pageTarget type="normal" id="pagetarget000007" value="7" playOrder="7">
      <navLabel>
        <text>7</text>
      </navLabel>
      <content src="part0000.html#page-7"/>
    </pageTarget>

The catch? While the OPF spec (part of ePub) says reading Systems must support NCX. (and mentions pageList offhandedly), it’s not at all clear how much of NCX is supposed to be supported. This ambiguity has meant that no reading systems (to my knowledge) have implemented support for pageList, so it’s appeal is primarily aesthetic. Unlike page-map, adding a pageList won’t make your ePub invalid. The EPUB Standards Maintenance Working Group is trying to clarify the NCX issue.

¹ What percentage of people actually have both media in front of them at the same time? It’s unquestionably better when trying to tell someone on the phone about the hilarious double-entendre on page 294, but I wonder how often this happens as well. Why not tell them to search for would you like to check my figures?. Finally, in STM & educational content, where this sort of thing might come up often, the headings are often numbered, and serve as a better guidepost (because they don’t rely on the same trim/pagination for various international editions, etc, etc).

Here’s a roundup of upcoming talks at various publishing conferences:

Digital Book World

January 27, 2010

When digital books account for 10% or more of revenue, indifferent third-party conversions aren’t enough. I’ll be presenting a plenary session on taking ebook quality control to the next level.

O’Reilly Tools of Change

February 22-24, 2010

Networked, Mobile & Landlocked: Current Ereaders

Keith Fahlgren and I will reprise our session from last year showcasing various ereading devices. We’ll focus on new devices released this year, including the Kindle DX, Nook and 3G Sony Reader, and describe the current and future impact of Android on mobile reading.

Open Publication Distribution System

Hadrien Gardeur (Feedbooks), Keith Fahlgren (O’Reilly Media, Inc. ), Liza Daly (Threepress Consulting Inc.)

You may have heard about OPDS and BookServer. This technical workshop will describe the OPDS format, its role in the larger BookServer architecture, and how to create and distribute OPDS book catalogs.

Reading and Purchase on the Open Mobile Web: Introducing Ibis Reader

This will be a pecha kucha/O’Reilly Ignite. I’ve never presented one before, so that should be, uh, fun?

BookNet Canada Tech Forum

March 25, 2010

This presentation will focus on the formatting options available in ePub, how open standards like CSS are implemented in the real world, and identify those features of ebooks which can be controlled — and those that are just part of the new publishing reality.