Creating PDFs for EBrary Reader

Turns out that Brown has a subscription to at least part of the Ebrary archive of online books. I found this out because I was looking for a book in our library, using the online catalog, and it turned out that said book was available online. Very cool!

Well, kind of cool, until I found out what this involved. By default, you basically have to read the document online. You can download the "EBrary Reader", which is a Java application, and read documents using it. But it's kind of clunky, to say the least. What I wanted was a PDF that I could then use as I wanted. How to get one?

I noticed that the EBrary Reader would allow me to print, so I thought maybe I could print the file as a PDF, since the default print dialog for Fedora lets you do that. Unfortunately, the Java application was not using the system dialog, but a Java dialog, so that didn't work.

A little googling led me to the cups-pdf package, which installs a system-wide PDF printer for the Common Unix Printing System. A quick "sudo yum install cups-pdf" was enough to give me access to that.

The next step was to convert this file to DjVu, which tends to be much smaller than the corresponding PDF. I've done this a million times before, so figured it would be pretty easy. Unfortunately, it was not.

The first step was to run the pdfimages command (from the poppler-utils package):

pdfimages -p file.pdf p

to extract the page images. Imagine my surprise when I got 420 images from a 20 page paper! It turned out that each page was constructed from 21 different images, stacked on top of each other. (To help with download times?)

Fortunately, I've had enough experience with ImageMagick to know this was not a problem that could not be solved. It took a little more googling, and a little experimentation, but eventually I found out that:

for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20; do
  montage p-0${i}*.ppm -geometry +0+0 -background none -tile 1x21 page-$i.tiff;
done

would stack all the images back on top of each other.

So now I had 20 page images, all as TIFFs, and those could then be fed to ScanTailor for processing on the way to creating a DjVu.

EBookDroid

For a while now, I've been trying not to print out copies of papers I'm using for class. I had a Kindle DX a while back, and I tried loading PDFs of papers into it. But the page rendering was so slow it was painful, and a real problem in class itself, since finding a particular passage would simply take too long.
More recently, I've acquired a Samsung Galaxy Tab 10.1, which makes for an excellent e-reader. It has a nice, bright screen, good resolution (1280x800), and lots of memory. Plus, I can access my course web site, or JSTOR, or whatever, directly from it (and with tools like AndFTP, I can even access my servers using SFTP and public key authentication).
But as I've mentioned elsewhere, a lot of the material I've scanned for my own use is in DjVu format, so I recently found myself needing to find a DjVu reader for Android.
It turns out (unsurprisingly, really) that there are several options, but the one on which I've settled for now is EBookDroid, a free and open source (GPL'd) ereader. Not only does it handle DjVu, it's got a lot of other nice features, such as the ability to set lots of named bookmarks (making particular passages easy to find). But one of the nicest things about it is that it will automatically split pages, if the file you are reading happens to have facing pages on a single page, like a photocopy. And it will automatically scale pages to the content area, not to mention wash your laundry and make your dinner. An excellent tool.
I've seen some people mention that EBookDroid "hijacks" PDF links from the browser, but I have not seen this issue myself. It does register itself as one possible PDF reader, but that is all.

Scan Tailor

For quite some time now, I've been putting all the readings (except books) for my various classes online, on my course websites (such as here or here). In the good cases, this just means downloading stuff from JSTOR or some equivalent, or linking directly to it, though I very often convert the PDFs I get that way to DjVu files, since (a) the DjVu files are often a lot smaller (like one tenth the size), and (b) I prefer to have files where two pages appear on a single page, so that when they are printed you don't waste so much paper.
I've developed a number of tools to make this easier to do. I'll blog about them later. (Some of them are already available here.) What I wanted to mention today is a really, really cool program I found last weekend, called "Scan Tailor". It's "free and open source" (GPLv3) and available for Linux, OSX, and Windoze.
I've found that the easiest way to scan stuff I need from books (e.g., Dummett's "Frege's Myth of the Third Realm", from Frege and Other Philosophers) is first to make a photocopy of the paper and then to scan that. (This is easiest with a sheet-fed scanner, such as the HP Scanjet 5590, which works fine with Linux.) You can simply make a PDF or DjVu from the result, but, if you just do that, it will usually look like crap. For one thing, you get big black marks along the sides and, often, down the middle (where the spine of the book is). And the pages can be hard to read on screen, since they are often not quite square. Even half a degree's rotation is easy to see, and very annoying.
So, in the past, I'd load the pages one by one into Gimp (an open source image editor) and fix them up. But this was time consuming, and hard to get right. That is where Scan Tailor comes to the rescue.
Here's how it works. You put all the page images into some directory, and then you open Scan Tailor and create a new "project", pointing the program at that directory. All the pages then get loaded up. You can then rotate them, if need be. But the really useful bit is that the program will then automatically (i) split the pages in half (which you need to do for the next step), (ii) deskew them (i.e., correct for rotation of the text), (iii) put a bounding box around the actual text (thus eliminating the black marks), (iv) put clean new margins around that text, (v) despeckle the images (remove stray black dots), and then (iv) output the resulting page images to a directory of your choosing, so you can assemble them into a PDF or a DjVu.
And at every one of those steps, you can intervene to make manual corrections, if need be. It totally, totally rocks.