The Tar Pit

A short introduction, for the atechnical: Lisp is one of the definitory programming languages for the fields of computer science and engineering. Common Lisp is a particular instance of Lisp, designed at some point in history as a unifying standard for various Lisp dialects. GPG is for the time being the one and only swiss army knife of cryptography, a tool that can aid one at identity (read: asymmetric key) management, encryption, sealing¹ and many others. Unfortunately, in the last few years the integrity of GPG itself has been a target for the usual wreckers and thus its days are numbered.

It very often happens in systems that two such seemingly unrelated components need to be interfaced; more specifically, in our case we would like to call GPG functionality from Common Lisp. "Traditionally" -- whatever that might mean -- this is supposedly achieved through the GPGME library, which includes a set of Common Lisp bindings.

I specifically added the "supposedly" above because I for one couldn't achieve this feat. While GPGME compiles without errors on my systems, the "language binding" glue code, written in Lisp and depending on CFFI, throws very weird type errors that I for one am not willing to debug, especially given the general lack of documentation on Google's interwebs. It may very well be the case that I am technically incompetent. But I am rather willing to stick my hand in the fire² that GPGME makes dishonest assumptions about the version of CFFI; and moreover, that both CFFI and GPGME (especially the latter) are balls of doesn't-fit-in-head that aren't worth half the attention that I gave them.

This blog post proposes a more elegant alternative for CL-GPG interfacing, a. that makes minimal assumptions about the underlying environment (e.g. Common Lisp implementation, GPG version), i.e. works on any "modern" Common Lisp implementation running inside a Unix system³; and b. whose implementation fits in a blog post, and thus in the reader's head.

First, the assumptions. We assume that the CL implementation can launch other Unix processes and communicate with them through pipes connecting the usual standard I/O streams (std{in,out,err}). The astute reader may have already figured out that this approach is not fundamentally different from the Bash/Perl approach of launching a new process, feeding it input, then piping the output back to our script or program. That is all there is to it.

For this purpose we will define a primitive lispy-run-program function, that can rely on whatever Unix process management primitives the underlying Common Lisp implementation provides. The example below uses UIOP's run-program function.

(defun lispy-run-program (command &optional (input ""))
  "Run `command' with optional `input'.

Returns two strings, representing what the command wrote on the standard
output and standard error respectively. The result (error code) of the
command is ignored."
  (let ((stdout (make-string-output-stream))
        (stderr (make-string-output-stream)))
    (with-input-from-string (stdin input)
      (uiop:run-program command
                        :input stdin
                        :output stdout
                        :error-output stderr
                        :ignore-error-status t))
    (values (get-output-stream-string stdout)
            (get-output-stream-string stderr))))

lispy-run-program takes as parameters a string command (the command to be run) and an optional string input (the input to be piped to the child process's standard input). It creates three (one input and two output) Common Lisp streams: stdin, stdout and stderr, and passes them to run-program. Then it "gets" the strings from the two output streams and returns both of them. Also note that the return code of the child process is ignored -- this can be changed by the programmer howsowever he or she wishes.

Let's try out a few invocations of lispy-run-program:

; Listing /etc/sh*
CL-USER> (lispy-run-program "ls /etc/sh*")
"/etc/shadow
/etc/shells
"
""

; Listing a non-existing file
CL-USER> (lispy-run-program "ls /bla")
""
"ls: cannot access '/bla': No such file or directory
"

; The equivalent of "echo 1 2 3 4 | cut -d' ' -f3"
CL-USER> (lispy-run-program "cut -d' ' -f3" "1 2 3 4")
"3
"
""

As seen above, this simple function puts a very powerful composition tool directly at our fingertips. We will use it to interface with the following set of GPG functionalities in Common Lisp: encryption, decryption and signature verification. Note that all our processing will be performed on plain-text ASCII-armored files, though the functions in this post can in principle be adapted to use binary inputs/outputs.

Looking at the GPG man page, and judging by the known behaviour of GPG, we notice that we can run it outside of a terminal-based environment -- and thus obtain full control over its input and output -- by passing the --no-tty argument. Thus, encryption can for example be written as:

(defun gpg-encrypt (input recipient)
  "Encrypt `input' to `recipient'."
  (lispy-run-program
   (format nil "/path/to/gpg --no-tty -ea -r ~a" recipient)
   input))

Where /path/to/gpg is the path to the GPG executable inside the Unix file system. Similarly, decryption:

(defun gpg-decrypt (input)
  "Decrypt ASCII-armored `input'."
  (lispy-run-program "/path/to/gpg --no-tty -d" input))

Similarly, clearsigning (not-quite-properly named "seal" here):

(defun gpg-seal (input &optional (uid ""))
  "Make a cleartext signature of `input'.

If `uid' is provided, the input will be signed using that specific key."
  (let ((uid-flag (if (string/= "" uid)
                      (format nil "-u ~a" uid)
                      "")))
    (lispy-run-program
     (format nil "/path/to/gpg --no-tty --clearsign ~a" uid-flag)
     input)))

Notice that gpg-seal gets an optional uid parameter that can be used to sign as a specific key (using GPG's -u flag).

Finally, seal verification:

(defun gpg-verify-seal (input)
  "Verify cleartext-signed `input'."
  (lispy-run-program "/path/to/gpg --no-tty --verify" input))

Now let's test our functions. First, we create a test key:

$ gpg --gen-key

# usual GnuPG key generation follows
# ...

gpg: key 48BECFE5 marked as ultimately trusted
public and secret key created and signed.

pub   4096R/48BECFE5 2018-04-23
      Key fingerprint = 533B 174D 1962 36B1 C066  670F CDD8 A167 48BE CFE5
uid                  Tarpiter 
sub   4096R/CF9F3670 2018-04-23

And now back in the Lisp console, we play with the four functions. First, encryption and decryption.

CL-USER> (defvar *secret*)
*SECRET*
; Encrypt the string "a very sikrit test" to tarpiter's key and store it
; in `*secret*'.
CL-USER> (setq *secret* (gpg-encrypt "a very sikrit text" "tarpiter"))
"-----BEGIN PGP MESSAGE-----

hQIMAzPNTIDPnzZwAQ//X+f2PGhEbouCYSxtaOXfY4XQU6vFjlts63PEM+5qD5b3
dSPVVJ+wjLZOK97eB0WGAqnGHRmIxpjr1vHVwK/b/uh+KPWTrCLbWZscGoSIUxrG
GJpsQLmRJx4NEStDhjZ1+PwFod0aHqFJ32chiP4bTQfKt1vLDi0Cs7eEDAZzaBXV
UrEa2t+IfY6BpOn4SUfMPJZDBnK+b1n7QV0gpRCp+x1qrf4Sun/PD7PuUy9zy66i
+HcVWnju4tEdpA7sNV/nsHHUzaDlVriJkFOTDNOvs4n8ku7Zvcl6GQjPFvrFYRo3
e/G7+1GxphyWZN6ARd3PNybMAopZ+FlhwYHIvfGJgt8E3p9HgrHR18RQt1WadJF1
NAYlVcF2LkY8qj4wdGwdhHnbDhuu8w+OsTF+RZS7c5FbtEKYaUgIT1lhiU5iPpkL
tzwbVXNn6VOCcG3O65yaxAxf6RFO1vSo1hcB0xRDBJeMhhBeU+bJt9mVIQDV7FKt
2mZQstZXOOFLT3CfIE4A0Nbe4F6/KYg+tImptnWtG2XP2a4RR39D4uAXV5mHpyDI
FkcmyK7Qw2gfVV5slxmRZdzna3dkCe9MQUZLj0oD5sAXAJlXroBWOG0yyJHKCgnN
AEhq4HEksMrQLkAVmpMIw4pYSLwxI6rlIJdWqDGUJICYj6qRh/OQPA9p0dclZJ/S
TQH9UnRvNzfdCaitQrEndwe1l7BjZtHJYAT/3AFVTJU4nC9kSNFqLVYH71itGEr8
5RhiNKHkdbNzb6HLmmuiZoQVzy+C60ofBy+F2ilR
=s/mJ
-----END PGP MESSAGE-----
"
; Decrypt the encrypted content of `*secret*'.
CL-USER> (gpg-decrypt *secret*)
"a very sikrit text"
"gpg: encrypted with 4096-bit RSA key, ID CF9F3670, created 2018-04-23
      "Tarpiter "
"

Then, sealing and verification:

CL-USER> (defvar *clearsigned*)
*CLEARSIGNED*
; Clearsign the string "a very authentic text" and store it in
; `*clearsigned*'.
CL-USER> (setq *clearsigned* (gpg-seal "a very authentic text" "tarpiter"))
"-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

a very authentic text
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJa3jHGAAoJEM3YoWdIvs/leKYQALexMI0Md83qYwnvcjnxmqSw
H7hGJWGir8UCBtR3nmzVUTj3dybwclwpkeAdbCMPlwfOuLtVlEi3u122t1fphq8g
TJw+2OMWeX0A5TdhD/t9ZXXy9yS7QjOgEyRhzORO+Lsb4lPf7FOdKKXLpFfihmkv
7Ew8IXoEUmvgcb0bEs5dX4Y7DKt6M9v0xdpUb9qrRa+Cp6qqwIpc+FCKq58dMGyA
SL41hswKCXSnhrdVVi97QzJKyj3QJ33OdyNfXjM12AbFEDHQHkYYtKYk5DiH82lS
lRVAIOwpD4xu7DdbUTRQJcQcBTJlN8RDCjwRx9E++ebzi/Mm5+8nXiiVyp2EQ4nk
AiUEoTIoWUAouPVzj9fdxijejKx/yQmRegRR9drTJkZ3cX5YYbiiILMll0gNPBYw
z4YMweLlpf6OyqdtlgwYym8nWEGOYmDA468h4NiEdzwaFEgoi5gzL5abHYnhoNLx
q+GVHE5imblo4to/iDnv6lhVKcU4IwhDzw9Ku9WtaXa8gOYcdKyru5PQm1EogpCG
kABFlikrvtPjg7RCJBpR1m/EmQW4t1BeOLyTk7Sc5RVimHc2V7C6cSmATHeDlLIY
qWYafbdRSo/b+bbHU9c47KSlaYhpElbmY7fj3uv6dUvKMBSo+i+8U3BBeNXF5O5N
zNIemhVEmp8cnnGY9Jf9
=B6SZ
-----END PGP SIGNATURE-----
"
; Verify the content and the seal in `*clearsigned*'.
CL-USER> (gpg-verify-seal *clearsigned*)
""
"gpg: Signature made Mon 23 Apr 2018 22:19:34 EEST using RSA key ID 48BECFE5
gpg: Good signature from "Tarpiter "
"
; Modify a character in the clearsigned payload.
CL-USER> (aref *clearsigned* 62)
#t
CL-USER> (setf (aref *clearsigned* 62) #c)
#c
CL-USER> *clearsigned*
"-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

a very authencic text
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJa3jHGAAoJEM3YoWdIvs/leKYQALexMI0Md83qYwnvcjnxmqSw
H7hGJWGir8UCBtR3nmzVUTj3dybwclwpkeAdbCMPlwfOuLtVlEi3u122t1fphq8g
TJw+2OMWeX0A5TdhD/t9ZXXy9yS7QjOgEyRhzORO+Lsb4lPf7FOdKKXLpFfihmkv
7Ew8IXoEUmvgcb0bEs5dX4Y7DKt6M9v0xdpUb9qrRa+Cp6qqwIpc+FCKq58dMGyA
SL41hswKCXSnhrdVVi97QzJKyj3QJ33OdyNfXjM12AbFEDHQHkYYtKYk5DiH82lS
lRVAIOwpD4xu7DdbUTRQJcQcBTJlN8RDCjwRx9E++ebzi/Mm5+8nXiiVyp2EQ4nk
AiUEoTIoWUAouPVzj9fdxijejKx/yQmRegRR9drTJkZ3cX5YYbiiILMll0gNPBYw
z4YMweLlpf6OyqdtlgwYym8nWEGOYmDA468h4NiEdzwaFEgoi5gzL5abHYnhoNLx
q+GVHE5imblo4to/iDnv6lhVKcU4IwhDzw9Ku9WtaXa8gOYcdKyru5PQm1EogpCG
kABFlikrvtPjg7RCJBpR1m/EmQW4t1BeOLyTk7Sc5RVimHc2V7C6cSmATHeDlLIY
qWYafbdRSo/b+bbHU9c47KSlaYhpElbmY7fj3uv6dUvKMBSo+i+8U3BBeNXF5O5N
zNIemhVEmp8cnnGY9Jf9
=B6SZ
-----END PGP SIGNATURE-----
"
; Verify it again.
CL-USER> (gpg-verify-seal *clearsigned*)
""
"gpg: Signature made Mon 23 Apr 2018 22:19:34 EEST using RSA key ID 48BECFE5
gpg: BAD signature from "Tarpiter "
"

and so on.

Finally, some considerations. First, the functions presented above assume that the recipient and uid arguments -- or for that matter any other arguments that might be passed from the program to the command line -- are either trusted or pre-processed by the programmer. Consider the following common example:

; We seal to tarpiter, but in the process also execute an arbitrary
; command.
CL-USER> (gpg-seal "" "tarpiter; ls /etc/passwd")
"*snip'ed GPG output*
/etc/passwd
"
""

There is no way to avoid this other than by doing very strict parsing/sanitization on the inputs, which is outside the scope of this article. It's also worth mentioning that some classes of problems, e.g. deedbot authentication⁴, use (constant) programmer-defined sensitive inputs, and thus are not susceptible to this leaky abstraction problem.

A similar issue is related to passphrase processing, assuming that the programmer uses such keys. For this purpose, GPG provides the --passphrase* class of command-line flags, of which --passphrase-fd is the sanest, as it can map to an explicit communication pipe between the two processes; on the other hand, --passphrase should be avoided at all costs, as anyone with the ability to execute e.g. ps can see the command-line of processes in the system.

Additionally, we notice that our functions do not process the output in any way, relying on the programmer for e.g. error checking or automation of signature verification, the latter being crucial to the correct implementation of V.

Finally, the reader has probably noticed that there is a lot of functionality missing from these examples. There are many ways to use GPG; for example the programmer may need to verify detached signatures instead of the clearsigned variety that we've provided. It would be pointless and potentially dangerous to attempt devising a general interface to GPG using this approach, which is maybe the only aspect giving GPGME an advantage. That aside, the examples above can be extended without much hassle to work with most of the well-known GPG recipes.

Of course, I'm not the first to have thought of this. v.pl interfaces with GPG this way, albeit from Perl rather than Common Lisp. Moreover I've found out that Andrew has written a Lisp V that uses SBCL's run-program in pretty much the same way as described here. I'm just adding my report to the journal in the hope that it'll be useful to other people attempting similar things in the future.

Update 1: Trinque has brought to my attention, and phf has followed up, that sending command-line inputs via format is very bad practice. In particular UIOP provides a somewhat saner approach to shell command passing, via escape-sh-token and similar functions. For example we can have uiop:run-program take a list of tokens instead of a single string, e.g. ("/bin/ls" "-l" "/"). Our lispy-run-program then becomes:

(defun lispy-run-program (command &optional (input ""))
  "Run `command' with optional `input'.

`command' is a list whose first element is the name of the program to be
executed and the rest of the elements are each a command-line argument
to be passed to the program.

Returns two strings, representing what the command wrote on the standard
output and standard error respectively. The result (error code) of the
command is ignored."
  (check-type command list)
  (let ((stdout (make-string-output-stream))
        (stderr (make-string-output-stream)))
    (with-input-from-string (stdin input)
      (uiop:run-program command
                        :input stdin
                        :output stdout
                        :error-output stderr
                        :ignore-error-status t))
    (values (get-output-stream-string stdout)
            (get-output-stream-string stderr))))

and gpg-encrypt and gpg-decrypt are implemented as:

(defun gpg-encrypt (input recipient)
  "Encrypt `input' to `recipient'."
  (lispy-run-program
   (list "/path/to/gpg" "--no-tty" "-ea" "-r" recipient)
   input))

(defun gpg-decrypt (input)
  "Decrypt ASCII-armored `input'."
  (lispy-run-program
   (list "/path/to/gpg" "--no-tty" "-d")
   input))

A reimplementation of the GPG interface using UIOP is provided in gpg-uiop.lisp. The reader is encouraged to read them and try them out, e.g. against the examples above.

Update 2: Trinque and phf also comment that UIOP, or rather ASDF3's inclusion of said library, is generally ill-regarded due to the former's inclusion as an implicit dependency. An implementation of lispy-run-program using SBCL's own run-program is provided in gpg-sbcl.lisp for the reader to inspect. Note that the GPG functions using it remain the same as gpg-uiop.lisp. Also note that sb-ext:run-program is by default stricter than UIOP's, in that it requires the full path of the program to be executed, PATH (and other environment variables) requiring explicit specification.

Filed under: computing, lisp.
RSS 2.0 feed. Comment. Send trackback.