Pyblosxom now running as FastCGI via flup

As I suggested I might a few months back, I have recently got around to running this Pyblosxom blog as a FastCGI process. The increase in speed is really impressive. Unless you happen to visit during a particularly busy period, the first time you hit this site a persistent Pyblosxom process will be started and will then hang around to service your subsequent requests, making them nice and speedy. This is made possible with Allan Saddi's flup library, which lets Python WSGI applications talk to webservers via FastCGI or SCGI. I'll go through the details of my setup (which uses lighttpd as its webserver) for the sake of other people who want to do the same

There are basically two steps involved: writing a FastCGI script which wraps the Pyblosxom WSGI app up in flup, and then configuring lighttpd to map a given URL to this script.

Here's my FastCGI script (with modified paths):

#!/usr/pkg/bin/python2.4
import sys
import Pyblosxom.pyblosxom
from flup.server.fcgi import WSGIServer

sys.path.insert(0, "/path/to/pyblosxom_config_file")
sys.path.insert(0, "/path/to/pyblosxom/codebase")
app = Pyblosxom.pyblosxom.PyBlosxomWSGIApp()
WSGIServer(app).run()

Pay attention to the shebang line: /usr/pkg/bin/python2.4 will only work on NetBSD systems. /usr/bin/env python should work anywhere and is better form in general. The second call to sys.path.insert is only necessary if you have Pyblosxom itself sitting around in a directory somewhere rather than installed as a Python library in the site-packages directory. All if this is discussed on the Pyblosxom page here, where they write a more or less identical script, minus the use of flup.server.fcgi.WSGIServer.

Here's the relevant part of my lighttpd.conf:

$HTTP["host"] == "www.luke.maurits.id.au" {
    fastcgi.server = (
        "/blog" => (
            "script.fcgi" => (
                "bin-path"              => "/path/to/fastcgiscript.py",
                "socket"                => "/tmp/pyblosxom.sock",
                "check-local"           => "disable",
                "disable-time"          => 1,
                "min-procs"             => 1,
                "max-procs"             => 32,
                "max-load-per-proc"     => 4,
                "idle-timeout"          => 60
            ),
        ),
    )
}

This maps the URI /blog on the www.luke.maurits.id.au virtual host to to the script we wrote above. When a request for that URI hits the server, lighttpd will run the script and talk to it through the unix domain socket specified (/tmp/pyblosxom.sock in this case). Make sure that, wherever you choose to put your socket, your webserver has read and write permissions. Note also that the path should be relative to any directory that you have lighttpd set to chroot into. This instance of the script will die after 60 seconds ("idle-timeout" above) pass without any requests coming in. Extra instances will be run, up to 32 ("max-procs") in total, to accommodate heavy load. A new process is created when ever there becomes more than 4 ("max-load-per-proc") queued requests for each existing process.

As I mentioned, this set up is very noticeably faster than running Pyblosxom as a plain CGI process - as one would fully expect. So far I have had a few problems where a huge number of FastCGI script processes are spawned for no apparent reason, slowing the server to a crawl and resulting in timeouts. I'm not sure what has caused these, though logic dictates it must be a problem with lighttpd. These seem to be over, but if they persist I will have to look into getting something other than lighttpd to span the FastCGI processes. Hopefully this won't happen. Until it does, enjoy the faster browsing!