A few changes to Feedparse's HTTP requester (prelude to Feedbot)
This post introduces s-xml-feedparse-http-fixes:
So what do these changes slash "HTTP fixes" consist of, more precisely? Two pieces of code are being repaired -- one of the repairs being required by Feedbot, while the other is more or less an opportunity for making the code a tad clea[nr]er. The latter (simpler) one goes as follows:
+(defvar *http-request-timeout* 60) ... - (parse-feed-string (http-request-with-timeout url 60))) + (parse-feed-string (http-request-with-timeout url *http-request-timeout*)))
In other words, we're calling
http-request-with-timeout with an operator-configurable timeout setting -- as it should have been in the first place, but let's not go into too many details, lest I end up burning myself and applying ice on the wound.
The former (more important) change is along the lines of:
+(defun http-request-thread (url timeout) + (handler-case + (drakma:http-request url :connection-timeout timeout) + ;; Note: this will have to be revisited once we have a sane view of + ;; all the conditions returned by drakma. + (t (c) + (return-from http-request-thread (values nil c))))) ... and a bit later: (let ((req-thread (sb-thread:make-thread - #'(lambda (url timeout) - (drakma:http-request url :connection-timeout timeout)) + #'http-request-thread
This means that the thread we're creating now runs the code in
http-request-thread instead of an anonymous function; but more importantly, that the call to Drakma's
http-request is now guarded by a handler-case that, long story short, acts as a wildcard exception catcher.
Feedbot benefits from this change by making sure that all HTTP requests -- that, as we know from above, occur on a separate thread -- are deterministic, i.e. the CL run-time doesn't end up throwing exceptions at us if somehow a request times out or who knows what other condition out of the operator's control. I know, this isn't the most fortunate of implementation choices, but well... if I do the right thing(tm) and fix Drakma at this very moment rather than later, and then that other thing, and then the other one, then I might get to publish Feedbot in a decade or so. Let's say that the debt is manageable -- for now, but not for long.
Up next: Feedbot, part the first.