The Tar Pit

This post is part of a series on Common Lisp WWWism. Before continuing, please read prior discussions on the subject.

As mentioned in an older study (admittedly, a more superficial one) of Common Lisp web servers, Hunchentoot is a monster with many tendrils -- in truth, one of the very few CL-based HTTP servers used in production nowadays. On the surface it weighs about six or seven thousand lines of code, depending on whether we count the comments. In depth it exposes a complex architecture, with support for e.g. cookies and sessions and other usefuls, but most importantly, its interface offers a ton of flexibility in handling requests and distributing workloads.

I've previously found this to be too much for my needs and thus went for something simpler, at least on the surface. However, following feedback and further discussion, and in light of the Republican need for a proper HTTP server, I have decided to review what the CL ecosystem has on offer, and more precisely to have another good look at Hunchentoot.

This sounded easy enough at the beginning, especially that it wasn't the first time. Only a Hunchentoot genesis would need to start somewhere, and the problem is there are no less than 111 somewheres to start from if we're to count the so-called "versions" in the CHANGELOG file, and almost 600 patches, that is, also "versions", although for some unknown reason those are thought of as something else by the original authors. Having realized this, I did two things: firstly, I asked for input from Hunchentoot users in the WoT; and secondly, I put my hazmat suit on and took a walk through the depths of Hunchentoot history, so as to have it inform my future decisions.

Below lies (with bullets!) the result of this documentation, in the form of a summary that groups changes into various categories, e.g. bugfixing, robustness, API changes etc. Looking at the changelog in chronological order, we see:

• Version 0.4.0: First version that comes with SBCL support (among others). Changes up until then have to do mostly with stability and small features, e.g. chunked encoding, keepalive.
• Versions 0.4.x: Mostly bug fixing, e.g. in mod_lisp, some SSLism.
• Versions 0.5.x: RAW-POST-DATA behaviour changes, added flexibility.
• Versions 0.6.x: Bugs, robustness.
• Versions 0.7.x: Support compilation without SSL; support Win32.
• Versions 0.8.x: Robustness, bugfixing.
• Versions 0.9.x: More Allegro/SBCL/etc. support; more interface flexibility.
• Versions 0.{10-14}.x: Debugging support, bugfixing, robustness.
• Versions 0.15.x: CLISP support; added CL-FAD as dependency; bugfixing, small interface changes.
• Version 1.0.0: Architectural redesign, small fixes, improvements.
• Versions 1.1.x: Architectural changes, bugfixes, shawarma, e.g.: "safeguard measures against XSS attacks". Why does JavaScript sanity need to be encoded in the implementation of a HTTP server anyway?

Version 1.2 is where development on GitHub begins and code changes can be examined. Versions prior to 1.2.0 seem to have been lost.

• Versions 1.2.{0-9}: Many bugfixes, documentation updates; also SSLism, support for non-SSL mode, portability, cookie handling, multithreading.
• Versions 1.2.{10-16}: Functionality to get local address and port; also, bugfixes, documentation.
• Versions 1.2.{17-24}: Multithreading API changes; also, bugfixes, portability, documentation; support is added for new MIME type -- why does the list of MIME types need to be part of the code and not a config parameter?; also, robustness, code sanity.
• Versions 1.2.{25,26}: HTTP header handling; SSLism, code sanity.
• Versions 1.2.{27,28}: Documentation updates, bugfixing, code cleanup, IPv6 support, other new functionality, robustness.
• Version 1.2.29: Revert IPv6 changes.
• Versions 1.2.{30-38}: Robustness, bugfixing, documentation, support for "additional HTTP status codes" (RFC 6585), multithreading optimizations, support for ports allocated by the system. Phf is using version 1.2.35 for btcbase, which makes this the likeliest starting point for a genesis.

Beyond version 1.2.38, Hunchentoot seems to have gone vaguely "agile", in the "move fast and break things" fashion. This includes SSLism changes, reintroduced support for IPv6 (which seems to bump up the required usocket version) and other potential breakages. Thus, I'm not going to look further in that direction.

As a side note, my hunch was unfortunately not at all unfounded: although I have given myself ample time to look at this, it still hasn't been enough, and here I am, about halfway through the work. As a side note to the side note: although I will genesize nothing less than a working item, I also expect the item in question to be not at all pretty, and full of warts etc., although in principle -- in principle -- it should be fairly easy to cut e.g. SSLism out.

Filed under: computing, lisp.
RSS 2.0 feed. Comment. Send trackback.