Timothy Lee of Princeton’s Center for Information Technology Policy has published a fascinating article titled Studying the Frequency of Redaction Failures in PACER in which he describes an online database of 1.8 million PDF court documents. He used my CAM::PDF Perl library to automatically search all of the documents attempting to detect solid rectangles drawn over text. Then he manually examined a subset of those, discovering that many of them are cases of accidentally leaked personal information. He proposes that his technique should be used as a pre-processing filter to detect problematic documents before they are added to the database.

Technologically, his technique takes advantage of some interesting details of Adobe’s Portable Document Format (PDF). In a PDF, the visual elements are all layered on top of each other using document coordinates. My library can make a list of all of the elements and what position they are on the page in the document. His code tries to identify black rectangles and looks for text elements which occur at the same overlapping coordinates as that rectangle. His code is not perfect because it seems to detect text that is drawn intentionally on top of background rectangles, but better safe than sorry with sensitive documents.

I just fixed a bug where appending a big PDF to a small one (“big” in terms of number of internal objects, which correlates with page count or byte count but is subtly different) often went wrong because I generated object ID numbers by simply incrementing a counter in the small PDF. Sometimes, that counter matched IDs in the bigger PDF, but those IDs are supposed to be unique per document, so things went badly. This bug is now fixed by simply taking the max ID of the two docs as the new counter value before incrementing to make a new ID number.

I never stumbled on the bug in my own work because I always appended (or prepended) the smaller doc to the larger one for performance reasons. CAM::PDF v1.54 is on it’s way to CPAN as I type this.

I’m grateful to Charlie Katz of the Harvard-Smithsonian Center for Astrophysics for providing me with a simple test case that exhibits the problem!

perl -le 'print 32139 + 92176'

which yields the copy-pasteable result:

124315

There are possible faster solutions, like Google (try typing “20 + 20″ in your search bar — down arrow, select-all and copy) or Quicksilver‘s math mode on Mac. But Perl is in my muscle memory, and I can quick solutions like appending the results to a file without even thinking about it.

In the command above, I’ve included two flags: -l and -e. -l is shorthand to put a newline at the end of every print statement. -e tells Perl that the next argument will be a script to execute.

Stricter languages, like Java, make it dramatically harder to accomplish the same goals. This is intentional, because Java also allows sandboxing of untrusted code in the same code space (pretty much impossible in existing dynamic languages despite efforts).

But what do you do if there is a problem with the third-party code and you can’t change it? In Java, you may just be screwed. For example, the Apache River distributed computing SDK uses “throws java.rmi.RemoteException” as a hint about which server methods are allowed to be invoked from the client. But, oops, Android’s Dalvik VM omitted all of the java.rmi.* classes to save resources. So trying to load a class with that “throws” declaration causes a “NoClassDefFoundError”. Because there’s no way to inject a third-party class into the “java.*” packages space, and Android developers have not been willing to add this simple Exception to Dalvik, this effectively kills all reasonable hopes of using River on Android. The River folks even talked about extreme hacks of rewriting the classes on load, but that’s too impractical. The answer from Android fans in scenarios like this seems to be “Why do you want to use that library anyway?“

A dynamic language would have just created a stub for that exception if it didn’t exist. Is that better or worse?

The result is brilliant, but a little rough (ExtJS and EnvJS don’t play nice, for example). The documentation is currently insufficient, but just barely so. One or two examples of usage would be enough to reduce this to a half-hour learning curve.

I have no idea where the name comes from… Maybe it’s a pun in German or something?