Massive XML files are uncommon in the general web development world, where the primary roles of XML are either as configuration files, read only infrequently, or for interchange across the web, in which case the files are necessarily small. It’s rare to encounter XML measured in gigabytes or more; data at that level is usually stored in a relational database.

For that reason I find myself frustrated with many XML tools, even those ostensibly designed to handle large amounts of data.  Too often they don’t scale well or at least easily.  I don’t believe that scaling should be a black art that each individual developer needs to solve independently.  Unfortunately, in commercial products ease-of-use is a key bullet point and computationally-difficult problems are hard to summarize in a user’s guide.

I tend to recommend open-source software most strongly in two scenarios: for small projects with limited budgets and for large projects with unique challenges.  There simply isn’t going to be a one-size-fits-all application for most interesting publishing work.

This is one of many reasons I’m excited by Google’s willingness to open its Google Books archive to researchers:  Python is a first-class programming language in the Google ecosystem, and Google has a good track record of open-sourcing those internal tools with limited commercial value.  I expect a lot of interesting work to come out of that archive once it’s available.

Bookworm now has integration with one part of this API. The Book Information page (available from the table of contents for each Bookworm book), displays results from the Google Book Search service for that title and author.

How good are the results?

Frankly I’m disappointed. The metadata is often sloppy: description fields are sometimes nonsensical, there are numerous spacing errors in which words run together, and there is much more data available when you click through to the Google Books page than was returned by the API.

Nevertheless, I have decided to include the data in this single place per book, to help Bookworm users find print editions of their ebooks (especially for public domain books).

The identifier problem

This latest API is not the first that Google Books released, but it is the first that allows arbitrary search queries (such as for title and author name). The previous version only allowed searches by ISBN.

The ePub standard requires that ebooks be tagged with a unique identifier but does not specify what that identifier is. Obviously public domain and non-books don’t have ISBNs. Some publishers are assigning an ISBN as the ePub identifier, but using unique ISBNs for their digital editions. It would be nice if I could uniquely tie the ePub version of a book on Bookworm to its print counterpart (and leverage powerful Google features like searching that book content), but that’s not going to be possible when the editions have different ISBNs. Similarly it would be difficult to encourage users to buy a print version from Amazon or other retailers without running the risk of pointing to an older edition or one by a different publisher.