PAR: Packaging for Perl applications

What is the best way to distribute a GUI application to users?

The three main choices are via an installer, via a standalone executable or via source. These choices vary a lot across platforms. Windows prefers installers, especially .msi files. Macs are quite happy with .app files, which are usually shipped on disk images. Most Linux variants use installers (.deb and .rpm) but some prefer source (e.g. Gentoo).

What if that application is written in Perl?

Perl is not typically considered a GUI language, but it does have bindings for GUI toolkits including Tk, wxWindows, Qt and GTK. Perl can be useful in the GUI realm as a rapid-development foundation or simply to add a couple of dialogs to a mostly-background process. A great barrier for entry, however, is that most platforms do not bundle these GUI toolkits with Perl and some platforms do not bundle Perl at all. Perl itself is most often distributed via installers, but the add-on modules that usually accompany any sophisticated Perl project are typically delivered as source. This poses a problem for most Windows users and many Mac users for whom this is too low-level a task to be tolerated. Only in the sysadmin-rich world of Linux and other Unixes aresudo cpan install Foo commands routinely tolerated.

The PAR project attempts to to create a solution to bundling the myriad files that usually compose a Perl application into a manageable monolith. The initial effort was modelled closely on the JAR concept that has proven to be a success in the Java community. As such, PAR files are simply ZIP files with manifests. If you have PAR installed on your computer, you can write Perl code that looks like this:

#!perl -w
use PAR 'foo.par';
use Foo;
...

and if Foo.pm is enclosed inside the foo.par file, it will be compiled from that source. Even more interesting, you can say:

#!perl -w
use PAR 'http://www.example.com/foo.par';
use Foo;
...

which will cause the foo.par archive to be downloaded and cached locally.

You may have noticed the sticky phrase above “If you have PAR installed…” That is a catch-22 of sorts. PAR helps users to skip the software installation steps, but first they have to … wait for it … install software.

To get around this, PAR takes another page from the ZIP playbook: self-extracting executables. The PAR distibution comes with a program called pp that allows a developer to wrap the core of Perl and any additional project-specific Perl modules into a PAR file with a main.pl and a .exe header to bootstrap the whole thing. What this gets you (on Windows in this example) is something like a Perl.exe with all of its modules embedded inside.

Here’s a simple example. Consider your basic Hello World application

---- hello.pl ----
#!perl -w
use strict;
use Tk;
my $mw = MainWindow->new;
$mw->Label(-text => 'Hello, world!')->pack;
$mw->Button(-text => 'Quit', -command => sub { exit })->pack;
MainLoop;

On a Mac, you have to have Tk installed (perhaps via fink install tk-pm586 if you’re on Tiger) and X11 running (perhaps via open /Applications/Utilities/X11.app). When you do so and run perl hello.pl you get something like this:

helloworld.pl screenshot

Now, say you want to give this cool new application to other Mac users. Telling them to first install Fink, Tk and X11 just for “Hello, World!” is ludicrous. Instead, you can build an executable like so:

/sw/bin/pp -o hello hello.pl

That creates a 3 MB executable called hello that includes the entire Perl and Tk. Send it to a friend who has a Mac (and X11, since we used a version of Tk that isn’t Aqua-friendly) and they can run it. If I were to make a Windows version of this it would be even easier on end users — on Windows, Tk binds directly to the native GUI so even the X11 prerequisite is not required.

Another benefit is version independence. The executable above is built against Perl 5.8.6 on Mac OS X 10.4. It should also work well on 10.3 or 10.2, even though those OSes shipped with older versions of Perl, because every part of 5.8.6 that was needed for Hello World is included in the EXE.

If you download that executable, you can open it with any Zip tool. For example:

% zipinfo hello
Archive:  hello   3013468 bytes   689 files
drwxr-xr-x  2.0 unx        0 b- stor 23-Oct-05 14:21 lib/
drwxr-xr-x  2.0 unx        0 b- stor 23-Oct-05 14:21 script/
-rw-r--r--  2.0 unx    20016 b- defN 23-Oct-05 14:21 MANIFEST
-rw-r--r--  2.0 unx      210 b- defN 23-Oct-05 14:21 META.yml
-rw-r--r--  2.0 unx     4971 b- defN 23-Oct-05 14:21 lib/AutoLoader.pm
-rw-r--r--  2.0 unx     4145 b- defN 23-Oct-05 14:21 lib/Carp.pm
... [snipped 679 lines] ...
-rw-r--r--  2.0 unx    12966 b- defN 23-Oct-05 14:21 lib/warnings.pm
-rw-r--r--  2.0 unx      787 b- defN 23-Oct-05 14:21 lib/warnings/register.pm
-rw-r--r--  2.0 unx      186 t- defN 23-May-05 22:22 script/hello.pl
-rw-r--r--  2.0 unx      262 b- defN 23-Oct-05 14:21 script/main.pl
689 files, 2742583 bytes uncompressed, 1078413 bytes compressed:  60.7%

(Note: you may see that the file sizes don’t match. That’s because the EXE also contains the whole Perl interpreter outside of the ZIP portion. That adds an extra 200% to file size in this case.)

Is it fast? No, the file need to be unzipped prior to use (which happens automatically, of course). Is it compact? No, 3 MB for Hello World is almost silly. But is it convenient? Yes. And that is often the most important quality when shipping software to users.

An interesting consequence of this distribution model is that the executable contains all of the source code. For some companies this may represent a problem (with some possible solutions listed at par.perl.org) but it is also a benefit in that you may satisfy any GPL requirements without having to offer a separate source download.


An important note for Windows is that, thanks to ActiveState.com, you do not need C compiler to build Perl yourself. They provide an installable package which include Tk pre-built. See links on par.perl.org for pre-compiled installers for PAR.

iTunes Music Store contest

Apple is currently having a contest counting down to 500,000,000 iTunes songs sold. To publicize the contest, they exposed an XML representation of the current count of songs sold. Just out of curiosity, I wrote a Perl program that uses that count information to extrapolate the next winning times. Not too hard considering the simple XML structure:

<?xml version="1.0" encoding="utf-8"?>
<root>
<count name="curCount" timestamp="Thu, 07 Jul 2005 23:16:00 GMT">485066974</count>
<count name="preCount" timestamp="Thu, 07 Jul 2005 23:11:00 GMT">485060489</count>
</root>

The problem is made easier by a few CPAN libraries.

  • LWP::Simple – retrieves files via HTTP
  • CAM::XML – an XML parser and DOM representation
  • DateTime::Format::Strptime – a date/time parser

So, here are my extrapolations followed by the source code. It looks like there will be a winner about once an hour. Of course, that assumes that the purchase rate is typical right now (a Thursday evening).

Retrieving http://www.apple.com/itunes/external_counter.xml...
Rate: 23.19 songs/sec
Predicted count right now: 485173522 (Thu Jul  7 19:35:14 2005 CDT)
485,200,000   Thu Jul  7 19:54:15 2005 CDT
485,300,000   Thu Jul  7 21:06:07 2005 CDT
485,400,000   Thu Jul  7 22:17:58 2005 CDT
485,500,000   Thu Jul  7 23:29:50 2005 CDT
485,600,000   Fri Jul  8 00:41:41 2005 CDT
485,700,000   Fri Jul  8 01:53:33 2005 CDT
485,800,000   Fri Jul  8 03:05:25 2005 CDT
485,900,000   Fri Jul  8 04:17:16 2005 CDT
486,000,000   Fri Jul  8 05:29:08 2005 CDT
486,100,000   Fri Jul  8 06:40:59 2005 CDT
486,200,000   Fri Jul  8 07:52:51 2005 CDT
486,300,000   Fri Jul  8 09:04:43 2005 CDT
486,400,000   Fri Jul  8 10:16:34 2005 CDT
486,500,000   Fri Jul  8 11:28:26 2005 CDT
486,600,000   Fri Jul  8 12:40:17 2005 CDT
486,700,000   Fri Jul  8 13:52:09 2005 CDT
486,800,000   Fri Jul  8 15:04:00 2005 CDT
486,900,000   Fri Jul  8 16:15:52 2005 CDT
487,000,000   Fri Jul  8 17:27:44 2005 CDT
487,100,000   Fri Jul  8 18:39:35 2005 CDT
487,200,000   Fri Jul  8 19:51:27 2005 CDT
487,300,000   Fri Jul  8 21:03:18 2005 CDT
487,400,000   Fri Jul  8 22:15:10 2005 CDT
487,500,000   Fri Jul  8 23:27:02 2005 CDT
487,600,000   Sat Jul  9 00:38:53 2005 CDT
487,700,000   Sat Jul  9 01:50:45 2005 CDT
487,800,000   Sat Jul  9 03:02:36 2005 CDT
487,900,000   Sat Jul  9 04:14:28 2005 CDT
488,000,000   Sat Jul  9 05:26:19 2005 CDT
488,100,000   Sat Jul  9 06:38:11 2005 CDT
488,200,000   Sat Jul  9 07:50:03 2005 CDT
488,300,000   Sat Jul  9 09:01:54 2005 CDT
488,400,000   Sat Jul  9 10:13:46 2005 CDT
488,500,000   Sat Jul  9 11:25:37 2005 CDT
488,600,000   Sat Jul  9 12:37:29 2005 CDT
488,700,000   Sat Jul  9 13:49:21 2005 CDT
488,800,000   Sat Jul  9 15:01:12 2005 CDT
488,900,000   Sat Jul  9 16:13:04 2005 CDT
489,000,000   Sat Jul  9 17:24:55 2005 CDT
489,100,000   Sat Jul  9 18:36:47 2005 CDT
489,200,000   Sat Jul  9 19:48:38 2005 CDT
489,300,000   Sat Jul  9 21:00:30 2005 CDT
489,400,000   Sat Jul  9 22:12:22 2005 CDT
489,500,000   Sat Jul  9 23:24:13 2005 CDT
489,600,000   Sun Jul 10 00:36:05 2005 CDT
489,700,000   Sun Jul 10 01:47:56 2005 CDT
489,800,000   Sun Jul 10 02:59:48 2005 CDT
489,900,000   Sun Jul 10 04:11:40 2005 CDT
490,000,000   Sun Jul 10 05:23:31 2005 CDT
490,100,000   Sun Jul 10 06:35:23 2005 CDT
490,200,000   Sun Jul 10 07:47:14 2005 CDT
490,300,000   Sun Jul 10 08:59:06 2005 CDT
490,400,000   Sun Jul 10 10:10:57 2005 CDT
490,500,000   Sun Jul 10 11:22:49 2005 CDT
490,600,000   Sun Jul 10 12:34:41 2005 CDT
490,700,000   Sun Jul 10 13:46:32 2005 CDT
490,800,000   Sun Jul 10 14:58:24 2005 CDT
490,900,000   Sun Jul 10 16:10:15 2005 CDT
491,000,000   Sun Jul 10 17:22:07 2005 CDT
491,100,000   Sun Jul 10 18:33:59 2005 CDT
491,200,000   Sun Jul 10 19:45:50 2005 CDT
491,300,000   Sun Jul 10 20:57:42 2005 CDT
491,400,000   Sun Jul 10 22:09:33 2005 CDT
491,500,000   Sun Jul 10 23:21:25 2005 CDT
491,600,000   Mon Jul 11 00:33:16 2005 CDT
491,700,000   Mon Jul 11 01:45:08 2005 CDT
491,800,000   Mon Jul 11 02:57:00 2005 CDT
491,900,000   Mon Jul 11 04:08:51 2005 CDT
492,000,000   Mon Jul 11 05:20:43 2005 CDT
492,100,000   Mon Jul 11 06:32:34 2005 CDT
492,200,000   Mon Jul 11 07:44:26 2005 CDT
492,300,000   Mon Jul 11 08:56:18 2005 CDT
492,400,000   Mon Jul 11 10:08:09 2005 CDT
492,500,000   Mon Jul 11 11:20:01 2005 CDT
492,600,000   Mon Jul 11 12:31:52 2005 CDT
492,700,000   Mon Jul 11 13:43:44 2005 CDT
492,800,000   Mon Jul 11 14:55:35 2005 CDT
492,900,000   Mon Jul 11 16:07:27 2005 CDT
493,000,000   Mon Jul 11 17:19:19 2005 CDT
493,100,000   Mon Jul 11 18:31:10 2005 CDT
493,200,000   Mon Jul 11 19:43:02 2005 CDT
493,300,000   Mon Jul 11 20:54:53 2005 CDT
493,400,000   Mon Jul 11 22:06:45 2005 CDT
493,500,000   Mon Jul 11 23:18:37 2005 CDT
493,600,000   Tue Jul 12 00:30:28 2005 CDT
493,700,000   Tue Jul 12 01:42:20 2005 CDT
493,800,000   Tue Jul 12 02:54:11 2005 CDT
493,900,000   Tue Jul 12 04:06:03 2005 CDT
494,000,000   Tue Jul 12 05:17:54 2005 CDT
494,100,000   Tue Jul 12 06:29:46 2005 CDT
494,200,000   Tue Jul 12 07:41:38 2005 CDT
494,300,000   Tue Jul 12 08:53:29 2005 CDT
494,400,000   Tue Jul 12 10:05:21 2005 CDT
494,500,000   Tue Jul 12 11:17:12 2005 CDT
494,600,000   Tue Jul 12 12:29:04 2005 CDT
494,700,000   Tue Jul 12 13:40:56 2005 CDT
494,800,000   Tue Jul 12 14:52:47 2005 CDT
494,900,000   Tue Jul 12 16:04:39 2005 CDT
495,000,000   Tue Jul 12 17:16:30 2005 CDT
495,100,000   Tue Jul 12 18:28:22 2005 CDT
495,200,000   Tue Jul 12 19:40:13 2005 CDT
495,300,000   Tue Jul 12 20:52:05 2005 CDT
495,400,000   Tue Jul 12 22:03:57 2005 CDT
495,500,000   Tue Jul 12 23:15:48 2005 CDT
495,600,000   Wed Jul 13 00:27:40 2005 CDT
495,700,000   Wed Jul 13 01:39:31 2005 CDT
495,800,000   Wed Jul 13 02:51:23 2005 CDT
495,900,000   Wed Jul 13 04:03:15 2005 CDT
496,000,000   Wed Jul 13 05:15:06 2005 CDT
496,100,000   Wed Jul 13 06:26:58 2005 CDT
496,200,000   Wed Jul 13 07:38:49 2005 CDT
496,300,000   Wed Jul 13 08:50:41 2005 CDT
496,400,000   Wed Jul 13 10:02:32 2005 CDT
496,500,000   Wed Jul 13 11:14:24 2005 CDT
496,600,000   Wed Jul 13 12:26:16 2005 CDT
496,700,000   Wed Jul 13 13:38:07 2005 CDT
496,800,000   Wed Jul 13 14:49:59 2005 CDT
496,900,000   Wed Jul 13 16:01:50 2005 CDT
497,000,000   Wed Jul 13 17:13:42 2005 CDT
497,100,000   Wed Jul 13 18:25:34 2005 CDT
497,200,000   Wed Jul 13 19:37:25 2005 CDT
497,300,000   Wed Jul 13 20:49:17 2005 CDT
497,400,000   Wed Jul 13 22:01:08 2005 CDT
497,500,000   Wed Jul 13 23:13:00 2005 CDT
497,600,000   Thu Jul 14 00:24:51 2005 CDT
497,700,000   Thu Jul 14 01:36:43 2005 CDT
497,800,000   Thu Jul 14 02:48:35 2005 CDT
497,900,000   Thu Jul 14 04:00:26 2005 CDT
498,000,000   Thu Jul 14 05:12:18 2005 CDT
498,100,000   Thu Jul 14 06:24:09 2005 CDT
498,200,000   Thu Jul 14 07:36:01 2005 CDT
498,300,000   Thu Jul 14 08:47:53 2005 CDT
498,400,000   Thu Jul 14 09:59:44 2005 CDT
498,500,000   Thu Jul 14 11:11:36 2005 CDT
498,600,000   Thu Jul 14 12:23:27 2005 CDT
498,700,000   Thu Jul 14 13:35:19 2005 CDT
498,800,000   Thu Jul 14 14:47:10 2005 CDT
498,900,000   Thu Jul 14 15:59:02 2005 CDT
499,000,000   Thu Jul 14 17:10:54 2005 CDT
499,100,000   Thu Jul 14 18:22:45 2005 CDT
499,200,000   Thu Jul 14 19:34:37 2005 CDT
499,300,000   Thu Jul 14 20:46:28 2005 CDT
499,400,000   Thu Jul 14 21:58:20 2005 CDT
499,500,000   Thu Jul 14 23:10:12 2005 CDT
499,600,000   Fri Jul 15 00:22:03 2005 CDT
499,700,000   Fri Jul 15 01:33:55 2005 CDT
499,800,000   Fri Jul 15 02:45:46 2005 CDT
499,900,000   Fri Jul 15 03:57:38 2005 CDT
500,000,000   Fri Jul 15 05:09:29 2005 CDT

And the source:

#!/usr/bin/perl -w

use strict;
use LWP::Simple qw(get);
use CAM::XML;
use DateTime::Format::Strptime;

# Constants
my $url = "http://www.apple.com/itunes/external_counter.xml";

# Fetch, parse and extract tags from the XML
print "Retrieving $url...\n";
my $xmlstr = get($url) || die "Failed to retrive the XML";
my $xml = CAM::XML->parse(-string => $xmlstr) || die "Failed to parse the XML";
my @nodes = $xml->getNodes(-tag => "count");
die "Not enough data in the XML\n" unless (@nodes >= 2);

# Parse the times and counts out of the XML data
my $parser = DateTime::Format::Strptime->new(pattern => '%a, %d %b %Y %H:%M:%S %Z');
my ($count1, $count2, $time1, $time2);
foreach my $node (@nodes)
{
   my $timestr = $node->getAttribute("timestamp");
   my $timestamp = $parser->parse_datetime($timestr)->epoch;
   my $count = $node->getInnerText();
   if ($node->getAttribute("name") =~ /cur/)
   {
      $time2 = $timestamp;
      $count2 = $count;
   }
   else
   {
      $time1 = $timestamp;
      $count1 = $count;
   }
}
die "Failed to understand the XML\n" unless ($count1 && $count2 && $time1 && $time2);

# Compute the songs per second
my $rate = ($count2-$count1)/($time2-$time1);
printf "    Rate: %.2f songs/sec\n", $rate;

# Extrapolate to now
my $now = int($rate * (time()-$time2) + $count2);
print "    Predicted count right now: $now (".localtime()." CDT)\n";

# Extrapolate to other prize times
for (my $c=485000000; $c<=500000000; $c+=100000)
{
   next if ($c < $count2); # these have already passed...
   my $t = ($c-$count2)/$rate + $time2;
   # Insert commas
   my $cstr = $c;
   1 while ($cstr =~ s/(\d)(\d{3}(?:,.*|))$/$1,$2/);
   print "    $cstr   ".localtime($t)." CDT\n";
}

Firefox Extensions: Web Developer

This is the second installment of my exploration of useful Firefox extensions

One of the most popular and highly rated Firefox extensions is the Web Developer extension. This tool adds one or more of the following to every web page: a handy context menu or a toolbar. Both contain the same list of tools that let an experienced HTML coder investigate and manipulate the current page. The total list of features is too long to detail here (you really just have to try a few of them), but I’ll highlight some that I use personally.

My favorite feature is the one shown in the screenshot below. The “Validate Local HTML” uploads the page you are currently viewing to a validation service (by default, validator.w3.org). This permits you to test your HTML before uploading it to a public web server, or allows you to validate a page that is hidden behind a login or form submission.

A screenshot of the Web Developer context menu

Another handy feature for HTML debugging is the Outline control. This submenu allows you to specify any of a variety of HTML entities to highlight. The extension highlights them by drawing boxes around them.

The extension also has a few uses for general users. The Disable submenu has options to turn off aspects of annoying pages, like image animations, page colors or even entire styles for the worst-designed pages. The Forms submenu allows you to convert POST forms to GET, for cases when you want to tweak the form values in the URL bar, or bookmark the results of a form submission.

The Web Developer extension has quite an impressive array of tricks it can accomplish, all with standard operations. If nothing else it demonstrates how flexible and open a platform Firefox has become.

Next time: undoclosetab

Firefox Extensions: Introduction and Live HTTP Headers

Background

Interacting with the web has gotten more fun in the last year or so. Browser technological improvements seem to be getting a higher priority as Firefox, Safari and Opera are competing to present a superior, standards-compliant environment that benefits both developers and end-users. This means that developers can spend less time working around stupid bugs and more time creating interesting content.

These days, if you are building a web application a sound strategy is to create it initially under Firefox or Safari and then apply workarounds to IE problems at the end. It doesn’t really matter if you start with Firefox or Safari as anything that works under one is highly likely to work under the other (and any failure is more likely the developer’s fault, not the browser’s fault! yay!)

This is is marked contrast to five years ago when making a sophisticated website work in both Netscape and IE ususally meant deep sacrifices in functionality, or writing two separate websites.

Firefox in particular has caused an explosion of interest via its extension mechanism. Can you write Javascript and XML? If so, then you have the skills needed to customize the web experience. Because of this ease of development, the breadth of experimental new features has grown as the risk of trying out those features has shrunk. Of course, like the bulk of web itself many of the extensions are junk. But a few are winners that I can no longer imagine working without.

This post, and likely a few more that follow, will detail the Firefox extensions that I consider to be the most useful, primarily to web developers but also to general users. Eventually, I hope to also cover a few that are interesting, but not useful or good enough for regular use in my opnion.

And so, without further ado, I present…

Live HTTP Headers

Live HTTP Headers screen shot

This is the first Firefox plugin I ever installed and I still use it routinely today. Live HTTP Headers adds a new tab to the Page Info dialog that shows details of the HTTP request/response for that page. To see it, install the extension, visit a page and select Page Info (via the Tools menu, via the context menu, or via Cmd-I on Mac). Then select the Headers tab to see the HTTP headers panel.

This panel shows the full HTTP message exchanged between the browser and the web server, less the actual content. This data can be critical for debugging anything deeper than HTML/JS glitches. For example, I maintain the Perl module called CGI::Compress::Gzip which transparently compresses dynamic web content when the browser says that it can support it. I used the HTTP headers to determine what the browser was telling the web server regarding compression — in this case “Accept-Encoding: gzip” was the important one. Then I looked at the web server response, which would tell me the Content-Length (should be smaller when compressed!) and the “Content-Encoding: gzip” message.

The HTTP headers can also tell you what web server and platform is being employed at the other end. Finally, you get to see the raw cookie data that went over the wire.

On the command line, you can get easily see the response headers via something like

    wget -S -O /dev/null http://www.chrisdolan.net/

or

    curl -I http://www.chrisdolan.net/

but that’s not as real-world as viewing the browser transaction directly.

Viewing response headers can also be amusing. For example, visitors to Slashdot receive a random Futurama quote like the following:

    X-Bender: Farewell, big blue ball of idiots!

Next time: the Web Developer extension