-
Notifications
You must be signed in to change notification settings - Fork 4
ServerSideDesign
We deliver texts to readers from sources.
There are many sources of texts:
- Public domain sources like Project Gutenberg, Internet Archive, and Hathi Trust.
- Paid sources like Overdrive, 3M, and Axis 360.
- The electronic catalogs of other libraries. (Open Library is an example of this, albeit an odd example.)
- Library branch buildings.
These sources give us texts under more or less restrictive terms. They also give us bibliographic records--information about texts--under very liberal terms.
For some of these sources, we can keep a local copy of both the texts and the records (Gutenberg). For some we can keep a local copy of the records, but the texts are way too large to mirror (Internet Archive), or we may not have permission to mirror the texts (Overdrive). For some sources (other libraries' catalogues) it may not be possible to even keep a local copy of the records.
Some sources allow free access to texts. Some will only serve a DRM-encrypted text to a limited number of simultaneous readers, a limited number of times.
Some sources serve nice-looking EPUB documents. Some serve only PDFs. Some can serve either EPUB or PDF, but the EPUB always looks terrible and is full of OCR errors. Some sources only serve physical books.
Users want access to books. Of all the distinctions covered here, the only one that concerns them enough to affect the interface is physical vs. electronic. Users will also care about distinctions not covered here: the language a book is written in, and whether or not it's an audiobook.
Although we will be focusing on iOS devices initially, readers will expect to use all sorts of computers to get their books: iPhones, iPads, Android phones and tablets, Kindles, Nooks, Sony readers, Kobos, and ordinary computers.
The server implementation does the work of a library. It makes unprompted and prompted book recommendations. It can look up a bibliographic record. Through its interface the reader can check out or return a book. If a book is not available through the interface, the reader can put a hold on it and get it later--whether "get it later" means waiting a few months for an electronic version or walking to a branch library for a print version.
The server provides a uniform view to all sources of texts. Not all texts support all operations. An open-access source has no need to support "hold" or "return a book" because its inventory is not scarce.
If the server cannot mirror a source's metadata and keep it up to date, that source cannot be a full participant in book recommendations. "Metadata" here especially includes the currently available inventory. This means you, OverDrive. (It also means Open Library, and possibly Axis 360.) The best way for the server to keep its metadata in sync with the source is to listen to the source for notifications or to periodically fetch an event feed.
Some e-reading applications feature an enormous selection of books you can buy from an online bookstore. Public library buildings have a mediocre selection of books you can read for free. Our e-reading application has a truly awful selection of books you might be able to read for free.
Copying the "storefront" interfaces of other e-reading applications (as BiblioCommons does) will lead us to promise more than we can deliver. I think it's a good idea to copy the feel of a physical library (shelves with in-stock books on them and no markers for checked-out books), but I'd also like to introduce concepts from another source: Netflix.
Netflix's streaming service lets you watch videos for free, after paying a monthly fee. They have a problem: their selection is terrible. If you search for what you want to watch on Netflix, the answer is usually crushing disappointment or "how about ordering this on DVD?". And yet the Netflix streaming video service is very successful. Why?
- Their recommendation engine is really good at finding the subset of their awful selection that you are interested in watching. ("The Conversation, 1970s Gene Hackman thriller, never heard of it, but sure, looks good.")
- If something interests you, instead of committing to watch it right now, you can put it on a list and come back to it when you have time.
- They publish exclusive content that you can't get anywhere else.
3 is not really relevant to us (or is it?...), but I propose we learn from 1 and 2.
In one respect, our selection is worse than Netflix. While they have the rights to a title, an unlimited number of people can stream that title simultaneously. We can only offer a title to around 1-10 people simultaneously. The effective inventory we can offer a given patron is much smaller than our actual inventory, so finding the gems is even more important.
In one respect, our selection is better than Netflix. We have free access to the public domain. There are public domain books that are still very popular, and more appealing to readers than public domain movies are to Netflix users. But the vast majority of the public domain is awful, lacking even the appeal of novelty. Making sense of this cornucopia requires some way of finding the gems in the junk.
Conclusion: focus on feeds of recommendations. If someone does a search, the answer is probably going to be "how about checking that out of a branch library?" or "how about standing in line for 3 months?" So downplay searches as much as possible--just like physical libraries do.
A combination of OPDS, LCP, and plain HTTP will cover most of this.
- Give me a feed of recommendations (OPDS)
- Give me a feed of search results (OPDS+OpenSearch)
- Give me a preview of this book (HTTP)
- Give me this book (possibly with license) (HTTP or LCP)
- I want to extend my license for this book (LCP)
- I want to surrender my license for this book (LCP)
- Put this book on hold/put me in the hold queue for this book (undefined)
- Release my hold on this book/remove me from the hold queue for this book (undefined)
- Tell me about your entire inventory.
- Tell me everything that happened to your inventory recently.
- Give me a feed of search results.
- Give me a preview of this book.
- Give me this book licensed to this patron.
- Extend this patron's license for this book.
- This patron is surrendering their license for this book.
- Put this book on hold for this patron/put them in the hold queue.
- Release this patron's hold on this book/remove them from the hold queue.
- Something happened to my inventory (e.g. someone checked out a book, a book was added to inventory).
| Message from server to source | Overdrive | 3M | Axis360 |
|---|---|---|---|
| Tell me about your entire inventory | via Products link? | "Get Library Events" + "Get Library Purchase Count" | "Availability" with "updatedDate" to get historical information, then "Availability" to get information about specific books |
| Tell me what happened to your inventory recently | Search with lastupdatetime + follow-up with Library Availability? | "Get Library Events" | "Availability" with "updatedDate" |
| Give me a feed of search results | Search API | Not supported | Not supported |
| Give me a preview of this book | Linked to from book detail | *Not supported* | *Not supported* |
| Give me this book licensed to this patron | Checkouts API + Download API | "Check Out", but *no way of getting the actual book* | "DRM Create"/"DRM Setup" + "Checkout" |
| Extend this patron's license for this book | Not supported | Not supported | Not supported |
| This patron is surrendering their license for this book. | Not supported--this is a DRM operation | "Check In" (but DRM operation also required) | Not supported (probably a DRM operation) |
| Put this book on hold for this patron/put them in the hold queue. | Holds API/"Place a hold" | "Place Hold" | "Add to hold" |
| Release this patron's hold on this book/remove them from the hold queue. | Holds API/"Removing a hold" | "Cancel Hold" | "Remove Hold" |
| View all of this patron's holds and checkouts. | Linked to from patron representation | "Get Patron Circulation" | *Not supported* |
| Hear about checkins, checkouts, hold placements, hold releases, availability of new titles, license expiration (for the book), books becoming available to next person in queue. | DOESN'T HAVE (via inference): Books becoming available to next person in queue. Can _generally_ infer everything else, but don't really know (i.e. who gets the book next). | DOESN'T HAVE: Availability (to next person in queue), Hold Releases | Foo |
For our first test build, targeted for the end of July 2014, we will need a live server performing authentication, search, and delivery of our Project Gutenberg corpus.
This will include:
- User Model
- Authentication
- Reading events to keep on server - Things that they have right now
- App debugging information
- Navigation feed [in whatever language - i.e. books in language] [mostly statically generated]
- Recommended lanes -- not paginated
- Other feeds [OPDS acquisition feeds] will be paginated - next link to next page. Will follow OPDS pagination guidelines
- Sorts [OPDS "facets"] - By title, By Author - facet groups will be sent to client
- Search: [Title + Author] - might not be fast
This work temporarily puts on hold:
- Cover images
- Book Description
- Book Quality Assessment
- Permanence of database
- Hierarchy + quality of hierarchy
Authentication will be handled by connecting to the iii authentication server.
Leonard is investigating the viability of Heroku as an architecture for this server, but it might ultimately not make sense to keep it there.