# Pushing Haunt to Its Limits

December 12, 2022 ❖ Tags: writeup, programming, lisp, guile, scheme, webdev

When I began to pen this article, my intent was merely to describe a comment system I'd written in Guile. But as often happens when I write, my words escaped the scope I had originally intended, and I soon found myself recording the history of every line of code I've written that's ever been run by a web server. I settled on allowing this to be an article about incorporating dynamic content into a Haunt site – a use-case that Haunt probably wasn't built to support, but which works surprisingly well due to Haunt configurations being ordinary Scheme programs.

## A Reason to Demarcate "Dynamic" and "Static"

A comparison that's sometimes made in discussing personal websites is "dynamic" versus "static." To me, dynamic is something that requires server-side logic or rendering, and static is something that doesn't. A website built using Jekyll and hosted solely on Neocities or GitHub pages is static, at least by my definition, and a WordPress or Drupal blog is dynamic. The picture is muddied by the existence of third-party services like Disqus or utterances, but I digress. I make the distinction in this article because I have two physically isolated machines with distinct purposes: one is a static web server, and the other runs the software that's responsible for implementing the "dynamic" capabilities.

In particular, this is all firmly seated in my old man's home network setup, which he uses to host (among other things) a website, so he has a machine running Apache. The page you're viewing now is hosted on that machine, and so is my brother's website. We're able to host them all on the same machine thanks to name-based virtual hosts. On the same network is a separate Gentoo server that's running a couple of services for myself – because I don't like the idea of running experimental garbage on a server that belongs to my dad. Apache supports operation as a forward proxy, so I'm able to have dad's server mediate traffic between the WAN and my Gentoo box. Initially, I only used this for exposing my Pleroma instance to the internet, but I later added a handler for jakob.space/api/*, which hits the Guile application that this article is about.

## The Beginning: An RSVP System

Earlier this year, I wanted to have some folks over to celebrate my 22nd birthday. I did what anyone in this situation would do and wrote my own RSVP system.

If you're just inviting a dozen or so people over, and don't otherwise have industrial-grade requirements, it's a simple thing to write yourself. I think the database schema does well to summarize the workings of the system:

CREATE TABLE IF NOT EXISTS events (
id SERIAL,
title varchar(128) NOT NULL,
description varchar(16384) NOT NULL,
datetime timestamp with time zone NOT NULL,
location varchar(128) NOT NULL,
PRIMARY KEY (id)
);

CREATE TABLE IF NOT EXISTS invitations (
id SERIAL,
vanity char(12) NOT NULL,
created_on timestamp with time zone default current_timestamp,
capabilities bigint NOT NULL,
event_id integer NOT NULL,
PRIMARY KEY (id)
);

CREATE TABLE IF NOT EXISTS rsvps (
id SERIAL,
vanity char(12) NOT NULL,
invitation_id char(12) NOT NULL,
event_id bigint NOT NULL,
fullname varchar(128) NOT NULL,
email varchar(256) NOT NULL,
guests varchar(1024) NOT NULL,
attending varchar(32) NOT NULL,
PRIMARY KEY (id)
);


I wanted a way to communicate information about the event (where and when) to friends, and end up with a list of attendees so I'd know how many burgers and beers to get. The former is represented by the events table, and the latter is represented by the rsvps table.

The invitations table is a bit more interesting. Invitations are "magic links" containing a unique identifier (the value in the vanity column) for the person I'm inviting. When an RSVP is submitted, I keep track of which invitation code was used, so if someone's sharing invitation links with people I didn't invite myself, I know who to get mad at.1 Also, some people get more information than others. In this particular example, my friend was graduating on the same day, so we did a single celebration for the both of us. I wanted her to be able to see the list of attendees, but I didn't want anyone else to be able to see it, so I added a capabilities column. It's a bigint, and I use bit-flags to represent individual capabilities such as whether or not a particular invitee is able to view the guest list.

For the actual implementation of the RSVP system, I wrote some JavaScript to fetch the event info (via XMLHttpRequest), display it, generate a form, and when the user presses submit, send that off as a JSON object. It's a lot of uninteresting and unsurprising code, so I won't discuss it here, but you can find it here if you'd like to see how it works.

The JavaScript needs a server to interact with, which is where Guile comes into the picture. I chose to use the built-in (web server) module over a2 web framework like Artanis. The interface is simple to use: you call run-server on a lambda that takes a request as a parameter and returns a response. In this case, I have two endpoints, GET /api/rsvp/event-info and POST /api/rsvp, with different behavior, and I defer to some "handler" depending on which endpoint is being requested.

(use-modules (ice-9 match)
(srfi srfi-1)
(web request)
(web response)
(web server)
(web uri))

(define (not-found request)
"Build a (somewhat) descriptive response for a non-existent resource."
(values (build-response #:code 404)
(uri->string (request-uri request)))))

(define (handle-api-request request body endpoint)
"Route handler for the API server."
(let ((method (request-method request))
(args (uri-query (request-uri request))))
(if args
(format #f "~a ~a (~a) (~a)" method endpoint args originating-ip)
(format #f "~a ~a (~a)" method endpoint originating-ip)))
(match (cons (request-method request) endpoint)
...
(('GET "rsvp" "event-info") get-event-info)
(('POST "rsvp") post-event-rsvp)
..
(_ (lambda (. args) (not-found request)))))

(define (main-request-handler request body)
"Server entry-point; parse request' and defer to routing system."
(let* ((path-encoded (uri-path (request-uri request)))
(path (split-and-decode-uri-path path-encoded)))
(define-values (response resp-body)
(if (string= "api" (first path))
(handle-api-request request body (drop path 1))
(not-found request)))
(values response resp-body)))

(run-server main-request-handler)


Some minutiae have been scrubbed from the above snippet, like appending a Access-Control-Allow-Origin header to the response and rate-limiting endpoints. I won't annotate every part of the RSVP system, but to give you a sense of how Guile acts as the "glue" between the JavaScript code and the Postgres tables, here's the code for post-event-rsvp:

(use-modules (json)
(squee)
(srfi srfi-1)
(srfi srfi-9)
(web request)
(web response)
(web uri))

(define (valid-receipt-code receipt)
"Check database to see if receipt'."
(and (= (string-length receipt) (base64-length (%vanity-length)))
(positive?
(length
(exec-query conn "SELECT * FROM rsvps WHERE vanity = $1" (list receipt)))))) (define-record-type <rsvp-update> (make-rsvp-update-parameters) rsvp-update-parameters? (invitation-code rsvp-update-code set-rsvp-update-code!) (name rsvp-update-name set-rsvp-update-name!) (email rsvp-update-email set-rsvp-update-email!) (attending rsvp-update-attending set-rsvp-update-attending!) (guests rsvp-update-guests set-rsvp-update-guests!)) (define (params->rsvp-update params) "Parse params', an alist, into a <rsvp-update>'." (let ((res (make-rsvp-update-parameters))) (set-rsvp-update-code! res (assoc-ref params "update")) (set-rsvp-update-name! res (assoc-ref params "name")) (set-rsvp-update-email! res (assoc-ref params "email")) (set-rsvp-update-attending! res (assoc-ref params "rsvp")) (set-rsvp-update-guests! res (assoc-ref params "guests")) (if (any not (list (rsvp-update-code res) (rsvp-update-name res) (rsvp-update-email res) (rsvp-update-attending res) (rsvp-update-guests res))) #f res))) (define (update-event-rsvp params) "Handler for updating an RSVP to an event." (let ((params (params->rsvp-update params))) (unless params (panic "invalid form data")) (unless (valid-receipt-code (rsvp-update-code params)) (panic "invalid receipt code")) (exec-query conn "UPDATE rsvps SET fullname =$2, email = $3, attending =$4, guests = $5 WHERE vanity =$1"
(list
(rsvp-update-code params)
(rsvp-update-name params)
(rsvp-update-email params)
(rsvp-update-attending params)
(rsvp-update-guests params)))
(values '((content-type . (application/json)))
(scm->json-string
((receipt . ,(rsvp-update-code params)))))))

(define (post-event-rsvp request body)
"Entry point for RSVP create/update. We dispatch on the parameters."
(let* ((params (json-string->scm (utf8->string body))))
(cond ((assoc-ref params "update") (update-event-rsvp params))
...
(else (panic "invalid invite/update code")))))



I'm using guile-squee to reach out to the Postgres database. It's a nice library and a great poster child for Guile's dynamic FFI interface, but it's little more than a wrapper around libpq – not a high-level interface. This is where the choice to use Guile has been a bit rough around the edges: you're on your own for a lot of pretty common tasks in web development world, like generating database queries or validating that a request is well-formed. And that's precisely what we're doing here. When handle-api-request calls out to post-event-rsvp, we figure out whether we're updating a previously submitted RSVP or submitting a new one (the code for that case has been omitted in the interest of brevity). In update-event-rsvp, I have to ensure the parameters are well-formed, parse them into a record, and then interpolate those into a SQL query that I wrote myself.

At the point this code was written, I hadn't written a line of Guile in about a year, and I was thinking of this more as throw-away code rather than something I'd be writing a blog post about. I wouldn't hesitate to describe it as especially ugly. The comment system, being newer, does a marginally better job of showing off the ways that Scheme allows you to be clever and avoid boilerplate:

(use-modules (json)
(squee)
(srfi srfi-1)
(srfi srfi-19)
(srfi srfi-26)
(web request)
(web response)
(web uri))

(define-json-mapping <internal-comment>
make-internal-comment
internal-comment?
json->internal-comment <=> internal-comment->json
(id        internal-comment-id)
(name      internal-comment-name)
(subject   internal-comment-subject)
(email     internal-comment-email)
(comment   internal-comment-comment)
(url       internal-comment-url)
(publish-time
internal-comment-publish-time
"publish-time"
(lambda (x) (string->date x "~Y~m~d ~H~M~S.~N"))
(lambda (x) (date->string x "~Y-~m-~d ~H:~M:~S.~N")))
(reactions internal-comment-reactions))

"Internal function for querying the approved comments on a post

This interface exists for dynamically generating the comment view from Haunt."
(define (make-internal-comment~ . args)
(let* ((approved (first (take-right args 2)))
(approved (string->date approved "~Y~m~d ~H~M~S.~N"))
(reactions (last args))
(reactions (if reactions
'())))
(apply make-internal-comment
(append (drop-right args 2) (list approved reactions)))))
(let* ((query "SELECT id, name, subject, email, comment, url, approved, reactions
FROM comments WHERE slug = $1 and approved IS NOT NULL") (result (exec-query conn query (list slug)))) (map (cut apply make-internal-comment~ <>) result))) (define (get-comments request body) "API endpoint handler for querying for the comments on a particular post This is a wrapper around get-comments-by-slug'." (define (normalize-record record) (json-string->scm (internal-comment->json record))) (let* ((query-string (uri-query (request-uri request))) (params (if query-string (decode-form query-string) '())) (slug (assoc-ref params "p"))) (unless slug (panic "missing slug' query parameter")) (values '((content-type . (application/json))) (scm->json-string (list->vector (map normalize-record (get-comments-by-slug (car slug))))))))  Here, leveraging the fact that the result of exec-query is "close enough" to the parameters we would want to pass to the record constructor, and deferring to apply. Of course, this example has warts as well. I'm having to call normalize-record because, even though I've defined a JSON mapping for <internal-comment>, scm->json-string doesn't know how serialize the record – we can only serialize it if we call internal-comment->json. So, for every record, I serialize it into a JSON object and then immediately deserialize it into an alist – effectively to erase the type information and yield something that scm->json-string knows how to deal with. I have to call scm->json-string in this case because I'm dealing with a list of records. There might be a better way to do this, but I haven't figured it out yet! And I think that's a fair summary of my experience writing this in Guile. Unlike other languages that see a lot of use in the web development world, in Scheme, there isn't a clear-cut "best way" or "best library" to do these sorts of things. It provides you with all of the tools you'd need to do things in a way that's beautiful and easily-understood, but if you're in a rush, and too lazy to sit down and generalize your problem, the code that you end up writing can be a bit hard on the eyes. Had I not chosen Guile, the language I would have reached for is Rust, which has a nice ORM library (Diesel) for interacting with databases, and most Rust web frameworks will take care of parsing form data into a struct (and rejecting if the request is ill-formed.) There, you're afforded similar facilities for writing your own abstractions, but the community has given you some enough cookie cutters that you don't have to think about the basic things if all you want to do is write a web application. One last thing I'd like to mention: this isn't entirely specific to Scheme. Some things I complained about above, like the dance I had to do with serializing JSON, might be easier to deal with if I were using a statically-typed language, but I'm certain the rest of it comes down to Guile having a smaller community. In other words: I don't think this is the fault of Scheme or Guile. With that, I hope I've nerd sniped someone into making a great new web framework for Guile. But I think that's all there is to say about the RSVP system. It was an interesting-enough proof-of-concept for me to experiment with running more things on the server-side. ## The Sequel: A Picture Gallery Despite the rough edges in the code outlined above, from the perspective of my non-technical friends, the RSVP system worked flawlessly and I was able to throw a huge party. We ate a ton of food and drank a lot of beer. My friend Aaron was also nice enough to take the DSLR from me and take some pictures! I wanted to put them somewhere for everyone who went to be able to see, so I put them on my website and used the "magic link" approach again. CREATE TABLE IF NOT EXISTS galleries ( id SERIAL, vanity char(12) NOT NULL, title varchar(128), description varchar(4096) NOT NULL, datetime timestamp with time zone NOT NULL, PRIMARY KEY (id) ); CREATE TABLE IF NOT EXISTS images ( id SERIAL, vanity char(12) NOT NULL, title varchar(128), filename varchar(64) NOT NULL, thumb_filename varchar(64) NOT NULL, datetime timestamp with time zone NOT NULL, PRIMARY KEY (id) );  The "gallery" system has the same architecture as the "RSVP" system: there's some JavaScript to fetch info about the gallery and render it client-side, and there's some Guile code running on my server for the JavaScript to interact with. The only part that was different is that now I was dealing with images. (define (image-exists? file-name) (define (string/= a b) (not (string= a b))) (and (string/= file-name ".") (string/= file-name "..") (member file-name (scandir (%gallery-image-directory))))) (define (read-image file-name) (let* ((ext (string-downcase (last (string-split file-name #\.)))) (mime (cond ((string= ext "jpg") 'image/jpeg) ((string= ext "png") 'image/png) (else (error "Unknown MIME type."))))) (values ((content-type . (,mime))) (call-with-input-file (format #f "~a/~a" (%gallery-image-directory) file-name) (lambda (port) (get-bytevector-all port)))))) (define (get-image request body) (let* ((query-string (uri-query (request-uri request))) (params (if query-string (decode-form query-string) '())) (file-name (car (assoc-ref params "name")))) (unless (image-exists? file-name) (panic "invalid filename")) (read-image file-name)))  I have a legitimate complaint about the (web server) module – I cannot, for the life of me, figure out how to respond with a binary payload without reading the entire blob into memory first. This is a problem, because the photos coming off of the DSLR are massive and my server process was literally OOM'ing. Unable to resolve it in Guile, I eventually gave up and used Rust for the "hosting the images" part. use ascii::AsciiString; use std::fs; use std::path::Path; extern crate ascii; extern crate tiny_http; const BASE_DIR: &'static str = "/opt/gallery-images/"; fn get_content_type(path: &Path) -> &'static str { let extension = match path.extension() { None => return "text/plain", Some(e) => e, }; match extension.to_ascii_lowercase().to_str().unwrap() { "gif" => "image/gif", "jpg" => "image/jpeg", "jpeg" => "image/jpeg", "png" => "image/png", "pdf" => "application/pdf", "htm" => "text/html; charset=utf8", "html" => "text/html; charset=utf8", "txt" => "text/plain; charset=utf8", _ => "text/plain; charset=utf8", } } fn main() { let server = tiny_http::Server::http("0.0.0.0:8069").unwrap(); loop { let rq = match server.recv() { Ok(rq) => rq, Err(_) => break, }; println!("{:?}: {:?}", chrono::offset::Local::now(), rq); let url = rq.url().to_string(); let path = Path::new(BASE_DIR); let append = Path::new(&url); if let Ok(stripped) = append.strip_prefix("/static-ext/") { let path = path.join(stripped); if !path .canonicalize() .map(|x| x.starts_with(BASE_DIR)) .unwrap_or(false) || path.is_dir() { let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); let _ = rq.respond(rep); } else { let file = fs::File::open(&path); if file.is_ok() { let response = tiny_http::Response::from_file(file.unwrap()); let response = response.with_header(tiny_http::Header { field: "Content-Type".parse().unwrap(), value: AsciiString::from_ascii(get_content_type(&path)).unwrap(), }); let _ = rq.respond(response); } else { let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); let _ = rq.respond(rep); } } } else { let rep = tiny_http::Response::new_empty(tiny_http::StatusCode(404)); let _ = rq.respond(rep); } } }  I added a second VirtualHost for a /static-ext/ path, which hits a process running that Rust snippet instead of the main Guile program. It may have been more sensible to just store the images on dad's server, since they're static content, but I've got a much bigger disk attached to my machine and didn't want to fill up his with a couple hundred DSLR photos. That's all. The gallery was a simple extension of the code that I'd already written for keeping track of RSVPs. The whole file is under a hundred LOC. ## Comments That's a lot of words about two systems you're unlikely to ever interact with unless you're one of the half-dozen or so people I still hang out with. The more prominent change for regular readers is the re-introduction of a comment system. There have been comment systems on jakob.space (and its predecessors) in the past, but they predate the first commit in the blog repository, so I'm devoting a few sections to talking about them; the lessons learned (however few) from previous iterations are responsible for some of the decisions made in the current implementation. ### The Previous Self-Hosted Comment System Before Haunt, I used Hugo. And before that, I wasn't using a static site generator at all. From 2015 to 2018, my personal website was running on top of Flask and SQLite. It used to look like this. Of course, I didn't have an especially compelling reason for my website to be running on Python. This was when I was picking up Python for a second time, and I heard that "building a website with Python" was an option. Using Flask was merely my attempt to understand what that meant. What I developed was a basic content management system: I'd write my content in HTML, commit it to the database by copy/pasting it into an SQLite GUI, and use Jinja2 to shoehorn that into some hand-crafted HTML templates. This workflow is frankly better suited to a static site generator, which is why I eventually dropped my Flask codebase for Hugo. But picking up Hugo wasn't my first response to realizing that my choice of tech stack was overkill – what I did, instead, was take advantage of the power afforded to me. I developed a comment system. I tried my damned hardest to find the code for it, but when I pulled the repository from a backup, I was reminded of how bad my git hygiene used to be. commit 0ca36ee37903be95f915bddcd620f5e2786a67f7 (HEAD -> master) Author: jakob <[redacted]> Date: Fri Oct 27 17:12:54 2017 -0400 Last commit before redesign commit 914ca5965e77bfce2642fed7cc2d14ab265cc714 Author: jakob <[redacted]> Date: Sat Jul 22 20:25:33 2017 -0400 Redid showcases and blog format commit 39fa9c5760617931eae73bc9a57e9f1d60945024 Author: jakob <[redacted]> Date: Sun Dec 4 16:57:42 2016 -0500 Initial commit.  I'm convinced it's lost to time. What I do remember is that I wrote a crappy captcha system for it with Pillow; any form you put on the internet is inevitably going to get attention from many kinds of web spiders, so I did the bare minimum for taping it off. The algorithm was: • Pick n random characters • Write each character to a fixed-size canvas at a fixed x offset and a randomized y offset • Add some "noise" by drawing nonsensical lines across the canvas Intuition tells me that this isn't a great approach to thwarting bots, but I can't really conclude that it was ineffective because I have no data: in the time that it was actively deployed, no one had commented on any of my posts. Not even spammers. I don't recall exactly why I discontinued the comment system, but I know that it happened prior to my switch to Hugo – the first post I made that received significant traffic3 was made before the switch to Hugo, and if I had kept it to the EOL of the Flask codebase, I think I would have remembered receiving a comment or two.4 ### Webmentions There was no immediate replacement to the original comment system. For a while, my website was just the post archive and "about me" page. I eventually built a spiritual successor in my attempt at a Webmention integration. Though, this is functionally quite different: rather than being stored in a central database that I manage, Webmentions are distributed across web. Generally speaking, Webmention is "just" a protocol for indicating to a website that you've linked to it, or otherwise mentioned it elsewhere. There tends to be some metadata associated with the content of the "mention," such as the name and website of the author, and what kind of interaction it is (a response, a bookmark, etc.) Furthermore, there are services like webmention.io to handle the protocol on your behalf, making it a fairly enticing option for a static website. One could draw comparison to other services like Disqus which I am averse to as they require the user to load non-free JavaScript. Webmention, on the other hand, is an open standard, and services that interface with the protocol typically provide a readily-queried API. In the case of webmention.io, the service itself is free software. I first implemented Webmention support circa 2019 over a weekend when I was taking a break at my parents' summertime cottage. The protocol was getting some attention on lobste.rs and HackerNews at the time, so I figured it would be a fun project. I was able to figure out just about everything I needed to know from Aaron Parecki's article. If you dig through that page enough, you'll find the first (and only?) Webmention reply-to that I've sent. What I won't talk about here is the code I wrote to have Haunt generate a Webmention "outbox" for comments I wrote, because I don't think it's particularly interesting. If you're really curious, the source code is here. I'd prefer to talk about being a consumer of Webmentions. My first attempt at an "integration" was this: /* * webmention.js -- Fetch and display mentions from webmention.io. * Copyright © 2019 Jakob L. Kreuze [REDACTED] * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License as * published by the Free Software Foundation; either version 3 of the * License, or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see * <http://www.gnu.org/licenses/>. */ function buildApiUri() { const snip = /https?:\/\/.*?\//.exec(window.location.href); const page = window.location.href.substring(snip[0].length); // TODO: Build up from aliases, etc. const aliases = [page]; return Array.concat( ["https://webmention.io/api/mentions.jf2?"], aliases.flatMap((alias) => { const ref = "jakob.space"; return [ target[]=http://${ref}/${alias}, target[]=https://${ref}/${alias} ]; }).join("&") ).join(""); } function getData(url, callback) { if (fetch) { fetch(url).then(function(response) { if (response.status >= 200 && response.status < 300) { return Promise.resolve(response); } else { return Promise.reject(new Error(Request failed:${response.statusText}));
}
}).then(function(response) {
return response.json();
}).then(callback);
} else {
let xhr = new XMLHttpRequest();
callback(JSON.parse(data));
}
xhr.onerror = function(error) {

### On The Choice to Use Postgres

I already had a database instance for Pleroma. If it wasn't already there on my server, I would have considered SQLite.

### Fitting This Together with Haunt

For the same performance reason I gave for having the captcha behind a button that you need to click, I wanted to keep Haunt around. I also just think it's a great piece of software. This site is still generated by Haunt – there just happens to now be a separate component for implementing a bit server-side logic, and Apache will forward traffic to that separate component if it's a request for something besides the static content that makes up the majority of my site.

The nice thing about Haunt is that "sites," or configurations in Haunt, are just Guile programs. Because I chose Guile for the "dynamic" part of my website, I was able to import most of the code I'd written for my Haunt configuration. So far, I've only used this in one place: the server-side rendered comment form. I'm able to import the function I use in Haunt to add the navigation bar, footer, and CSS, and use that for the form, as well as all of the SXML utility functions I wrote.

To give a very underwhelming conclusion: being able to import Haunt stuff and use it to generate the SXML that's output by a Guile (web server) program is pretty cool.

## Footnotes:

1

Of course, the times someone's shown up uninvited, they didn't RSVP in the first place. In retrospect, it was a bit silly to think that this would work the way I expected it to.

2

The.. only?

3

According to Apache logs, at least. I don't use any sort of analytics. If you're really curious, it was my first post about Rust.

4

As an anecdote, I had folks reach out about the post via email, which I see this as far less convenient than filling out a comment form. Hence, I conclude that it's likely I removed the comment functionality by this point.

5

If you read Aaron's article that I linked above, you'll see that writing a Webmention is very technical. You need a website of your own, and enough patience to generate a bunch of complicated HTML, just to write a comment.