Some time ago I promised that I’ll write something about measuring efficiency of some Elisp code.
Now, my guess was that the string version will be faster for short templates (due to the overhead of creating buffers), but the longer the template, the faster the buffer version.
That was right, though for other reasons.
Let us first present the methodology. This is completely non-scientific, though I think still fairly accurate. I used the built-in benchmark Emacs library. It provides (among others) the benchmark command, which takes the number of repetitions and the form to run as arguments. There are also two macros: benchmark-run (the benchmark function is basically a fairly thin wrapper around that macro) and benchmark-run-compiled (which does exactly what it says on the tin – it compiles the given form first). I first ran the benchmark-run-compiled macro so that I didn’t measure interpretation overhead, but it turned out that in my case, the difference was very small. For the first test, I decided to use a template of
Lorem ipsum dolor sit {{{beep}}}.
with the values parameter equal to (("beep" . "amet")). So I ran this code:
(benchmark-run
100000
(expand-template
"Lorem ipsum dolor sit {{{beep}}}"
'(("beep" . "amet"))))
(benchmark-run
100000
(string-expand-template
"Lorem ipsum dolor sit {{{beep}}}"
'(("beep" . "amet"))))
and got these results:
(5.100522955 7 1.1810125730000038) (0.9608260630000001 2 0.3009825070000005)
(By the way, the compiled version turned out not to be really faster.) What to make of it? The returned list (which is displayed as a message with explanation if you decide to use the benchmark command) consists of three elements: the time (in seconds), the number of times garbage collection kicked in, and the time the garbage collection took (also in seconds). As you can see, the string version is about five times faster, both including and excluding GC.
Next, I took a longer version of Lorem ipsum – more than a hundred kilobytes, and about 15k words. I sprinkled a hundred beep‘s to replace and ran this
(benchmark-run 100 (expand-template super-long-template
'(("beep" . "amet"))))
(benchmark-run 100 (string-expand-template super-long-template
'(("beep" . "amet"))))
And here are the results:
(1.097874253 2 0.13561446199999994) (29.576419415 417 28.299135672000013)
Wow. Everything proceeded as I have foreseen – almost. I was right that the buffer-based expansion would be faster for long templates, but I have misjudged the reasons. The gain coming from using buffers instead of strings (and, unlike strings, buffers need not be reallocated each time sth is added to them) is there (the buffer version is slightly faster excluding GC), but the real difference comes from the fact that GC went off like crazy for the string-based version.
What’s the lesson from that? I guess it is this: if you operate on long strings in Emacs, especially if you mangle them a lot, do use buffers and not strings per se. OTOH, if you want to write efficient code, take various factors (like GC) into consideration, and run tests to see if your predictions about speed were right.
A final reminder: remember that if using the benchmark command non-interactively, you have to remember to quote the form you are timing! If you don’t do it, then, according to normal evaluation rules, benchmark will evaluate the result of its evaluation as many times as you give, which is probably not what you want – in our case, that would actually measure the evaluation time of a string constant. The benchmark-run macro, however, is a macro, and normal evaluation rules need not (and do not) apply here. (If you use benchmark interactively, you must not quote it – then benchmark would measure the time of evaluation of the quoted form, which is basically negligible, since it evaluates to the s-expression itself!)
Happy hacking!