November 28, 2011

Recent Mod Perl Woes

I would imagine that most of you (insofar as there are any of you) who read this most miserable of blogs do so with some kind of feed reader. And as you should: the code running this decrepit site has not been updated in almost five years—an eternity in the Internet era. One problem with running such an old site is that when things break, they break badly. That was precisely what happened this past weekend when what should have been a routine security upgrade of my webserver (Apache) instead took out my whole web presence.

And while I'd like to say I have fixed things, alas it isn't so. I have merely hacked them so that they are again functional; a fix, unfortunately, is still missing. For anyone experiencing issues recently with Apache and mod_perl, I have outlined the issue below. (That past sentence, by the way, should be taken as a sign for most—or all—of you to skip what comes next.)

To start, what were the symptoms? Due to various security vulnerabilities in recent months, I had updated Apache to its latest version (2.2.21). At the time, I was also running mod_perl 2.0.5 and Perl 5.10.1. After the upgrade, which went through without incident, I went to restart Apache, which is when the trouble began. After checking the httpd.conf syntax, the startup script froze up. A quick top revealed that the httpd process was rapidly jumping to 100% of the CPU and then sitting there until terminated manually. Somewhere, something was freezing up as Apache started up. Since I had not changed anything in the code or the configuration, I was mystified—and more than a little worried. When Apache won't turn on, things are quite a bit more fucked than usual.

The logs, moreover, revealed nothing at all—no errors, no warnings, not even a goddamn informational entry. Having spent almost a decade with these systems, I almost immediately recognized this insanely obscure issue as a likely mod_perl problem. Commenting out the the "perl_module" directive confirmed my suspicions—Apache started without incident in that situation. Problem is, this entire site is based on mod_perl; without it, there was no Realm.

My initial thoughts were that the new Apache wasn't working with Perl 5.10.1 and mod_perl. So I went on a lengthy process of moving to Perl 5.14 (which basically involved rebuilding hundreds of packages on my server). Hours later, nothing. Still the same problem.

Next step was to try tracing system calls to see where things were jamming up. I used FreeBSD's truss utility and discovered that the parent process was never forking; the fork command would be called followed by pgetgrp and then nothing.

After lots of googling, I found some hints that perhaps Perl 5.14 and mod_perl 2.0.5 did not work with threads enabled. This again led me down not one but two rebuilding wild goose chases—recompiling everything to not use threads and then recompiling everything to use Perl 5.12 (with threads). Still no dice.

Nearly four days into the bug hunt, I finally decided tonight that I would need to go line by line in the configuration file to see where mod_perl was choking. This lead me to discover that neither the PerlSetEnv nor PerlSwitches -Mlib directives were working as expected (I use PerlOptions +Parent to have a separate Perl interpreter and namespace for each virtual host). I finally discovered this old thread. Adding the <Perl> </Perl> blocks prior to the environment directives, however, broke the PerlSwitches -Mlib directive (since mod_perl was already instantiated by the <Perl> block).

The temporary solution: a two-stage hack. First, setup a sitecustomize.pl to replace the PerlSwitches -Mlib functionality. Then, add the empty <Perl> block and voilà—Apache will start once again. Clearly, this is not a permanent solution; among other things, it introduces hard coded hacks into what was supposed to be site neutral code.

But at the same time, I'm not sure finding an elegant solution to keeping a code base that is nearly a decade old alive is worth my time either. These sorts of infuriating bugs are a poignant reminder of both the bad old days (debugging excruciatingly obscure mod_perl errors) and why I actually need to find the time to move this site to a modern infrastructure. Hopefully, between now and then, I will not have to spend too much time with my dear old friend of the past decade—mod_perl.