2025-03-24 Speeding up operations on a remote server with reusing the ssh connection
I have a script which performs several operations on some remote server, issuing a number of ssh commands, more or less like this:
scp project.zip scp://some_server/~ ssh ssh://some_server rm -rf ~/project-old ssh ssh://some_server cp -r ~/project ~/project-old ssh ssh://some_server unzip -o ~/project.zip -d ~/project
This obviously works, but is not very optimized. One obvious bottleneck is that every ssh command needs to create a connection to the remote server, which takes some time. What if it were possible to open the connection once and then reuse it whenever the need arises?
Well, it turns out that not only is this possible, but much easier than I thought. In fact, in the simplest case you don’t even have to change the script at all! Just add something like this to the ~/.ssh/config file:
Host some_server HostName ... User ... IdentityFile ... ControlMaster auto ControlPath ~/.ssh/%r@%h:%p ControlPersist 1m
This way, the first ssh command will create a connection and each subsequent command will reuse it. The connection will close automatically after 60 seconds of inactivity. How cool is that? (In the case of my script, I was able to gain a speedup of about 50%!)
I am not 100% sure about the security implications of this solution, though – I don’t see any obvious security holes, but I’m not an expert on that. One thing that might look dangerous is the fact that the connection remains open for one minute after the script is done. If you want to explicitly close it instead of waiting, just say ssh -O
exit some_server.
2025-03-17 Bash script and passwords
Some time ago I needed to write a script which had to ask the user for a password. After a short research it turned out that there are a few options.
The most rudimentary one is probably the “read” built-in command. It has an -s parameter which does not echo the characters typed. One potential drawback of that is that it stores the read text in an environment variable. This means that if you subsequently use it with a command which is not another built-in, the password might be exposed in the output of ps -ef on some systems. You could echo it with redirection to a temporary file (preferably on a tmpfs in-memory filesystem) and then use the file in subsequent commands, and I’m almost sure this is safe, but please don’t quote me on that – I’m not a security expert, really.
Another interesting option I had no idea existed is the whiptail utility. It is surprisingly versatile – it allows us to get various data from the user (yes/no answers, menu selection, arbitrary text and a few more). It prints the user’s selection on standard output and uses the exit code to signal whether the user chose “OK”/”Yes” or “Cancel”/”No”, which I like better than environment variables. Go to the manual for the details.
Both the above options (and many others, like the dialog command-line utility) have one thing in common – they use the terminal the script is running in to read the password. This is fine in many circumstances, but not always. For example, if I run a script in a terminal, and half a second later this script asks me the password, it’s fine. However, if the script runs for 20 minutes and only asks for the password near the end, that’s much worse – most probably I won’t be just sitting there and looking at the terminal, ready to type a password whenever my machine wants me to do this. (Note that this is not at all unrealistic – for example, a long-running script could need to sudo something near the end.) Another possible (though possibly a bit contrived) situation is some kind of scheduled job (via cron, at or some similar tool). Yet another case (pretty common for us Emacs users) is a script run from within Emacs (like syncing emails with the IMAP server using mbsync). In some such cases ssh-agent takes care of getting the password from the user (and one day I’m going to learn how that works…), but some tools do not use it (Borg comes to mind).
Of course, it ssh-agent can do it, so can I, right? I even remembered the name of the tool ssh-agent uses under the hood – pinentry. Could I learn to use it?
The answer, of course, is “yes”. It turns out that you can use pinentry in a script, although its interface is a bit… strange.
First of all, let me mention that I could not find the “official” docs for pinentry on the internet, which is weird. The best I could find is some blog post which apparently contains the copy of the official manual. After a while it dawned on me: pinentry is part of GnuPG, and it contains… a TeXinfo manual!!! I’m a big fan of Info, so I immediately checked my system, and lo and behold – I already have the pinentry manual in Info!
The second interesting thing is that pinentry has a few “frontends”. For example, there is pinentry-gtk and pinentry-qt – but also pinentry-tty, pinentry-curses, and a bit suprisingly, pinentry-emacs. It is, of course, advisable to use the smallest pinentry you can – the larger the stack, the larger the potential attack surface. As much as I love Emacs, I have to agree with the pinentry manual:
Having Emacs get the passphrase is convenient, however, it is a significant security risk. Emacs is a huge program, which doesn’t provide any process isolation to speak of. As such, having it handle the passphrase adds a huge chunk of code to the user’s trusted computing base. Because of this concern, Emacs doesn’t enable this by default, unless the ‘allow-emacs-pinentry’ option is explicitly set in his or her ’.gnupggpg-agent.conf' file./
Unfortunately, even though I enabled the said option, I still wasn’t able to make pinentry-emacs work, even if just for the sake of an experiment.
The next strange thing about pinentry is the way you tell it what to do. I am accustomed to CLI tools which accept hundreds of command-line parameters to drive their behavior. It turns out that pinentry only has about a dozen (including --help, --version and --debug, which you wouldn’t normally use), and I would probably never use more than three of them. How do we use it, then?
Well, the design of pinentry is unlike most CLI tools. You start it with hardly any CLI parameters, and tell it what to do via its standard input. Conversely, pinentry’s responds via its standard output. The protocol it uses, called Assuan, is a bit complex, but for simple use cases like using pinentry to get a password from the user it is simple enough.
For example, this is what you would tell pinentry to set the “description” (shown above the password input box):
SETDESC Please tell me your secret
Other commands are, for example, SETTITLE (for the window title), SETPROMPT, SETOK and SETCANCEL (for the buttons), etc. – and most importantly, GETPIN to actually display the password window and return the typed stuff. The typed password is returned on pinentry’s standard output prefixed by the capital letter D and a space, so you need to parse that response (possibly along with OK’s from every other command like SETPROMPT) in the program you write to invoke pinentry. In the case of a shell script, you’d probably use sed for that:
sed -n 's/^D //p'
Of course, just filtering the output of pinentry through this won’t necessarily work if the user clicks “Cancel” – in such a case, there will be no line beginning with D and nothing will be returned, so the script should probably check for that. Also, if the password contains a percent sign, it is represented as %25 (and CR and LF are theoretically represented by %0D and %0A respectively, though they probably can’t be entered as a part of the password anyway). This means that you might want to e.g. use another sed command to take care of percent signs, for example like this:
sed -n -e 's/%25/%/g' -e 's/^D //p'
Also, if you run pinentry without the normal shell context, where you might not have your usual environment variables, you will probably want to use the --display option to tell it which X display to use.
I have to admit that I could not make some commands from the docs work – but the most important ones seemed to do what they should. So, from now on, I can make my scripts ask for the password in a much better way than just in the terminal they ran in (if they even ran in a terminal).
2025-03-10 Persisting variables across Emacs sessions revisited
You really never know. My post about persisting Emacs variables was a short tip, written in literally a few minutes to give me a respite from writing and time to complete some longer posts which are underway. It is no mystery that some posts are long (and I hope good) articles, and some are just short texts I put out when I’m overwhelmed with work – and the post about persist was definitely in the latter category. While I still consider it a useful tip, I never expected to get serious feedback about it – and here it is.
First, a commenter told me about the savehist feature, which has been part of Emacs since v22.1. I have to admit I didn’t know (or didn’t remember) about savehist – it’s no shame, though, Emacs is huge and nobody can be reasonably expected to know everything about it.
Anyway, savehist is something which indeed does roughly the same thing as persist. There are some differences, though. For example, persist saves the variables it knows about on killing Emacs, while savehist also saves them every 5 minutes (which is a nice thing, since Emacs crashes are very rare but not unheard of, especially if you run the bleeding edge like me). The main difference, though, is that savehist – as its name suggests – is really aimed at history variables. It uses a pretty clever trick to activate itself when the user enters the minibuffer, then checks if a history variable is used and if it is, it automatically adds it to its list of variables to keep. You can add custom, non-history variables for it to save with savehist-additional-variables and prevent some histories from saving with savehist-ignored-variables, so it can be used to keep any variable saved, though. (It also has a pretty interesting feature which makes sense for history variables but is absurd for most other uses, which is to automatically trim the saved list to a predefined number of elements – see the docstring for savehist-additional-variables.) My understanding is that this is aimed at users who want to customize Emacs to persist (some or all) history variables and possibly something else. On the other hand, persist is much smaller (about 120 lines compared to savehist’s 250) and hence simpler, and looks like something aimed also (or maybe even mainly) at package writers who want their package to have the value of some variable remembered across Emacs sessions. This is something that savehist explicitly doesn’t support – the list of saved and ignored variables are kept in user options, and I can’t imagine a package changing the value of a user option behind the scenes. With persist, the list of kept variables is not “centralized” in an option (it is kept in an internal variable persist--symbols, though) – every persisted variable’s value is saved to a different file, named after that variable, and you can just delete a file from the ~/.emacs/persist directory to stop persisting a no longer used variable. (Of course, if you have a persist-defvar invocation in your init.el, the file will be recreated when you restart Emacs.) This has an interesting consequence – if the variable’s name contains a slash, then its value is kept deeper in the directory hierarchy. (If you are wondering what would happen if someone used an actual null byte in a variable name – which is technically possible, of course – well, I checked that, and rather predictably you get a wrong-type-argument filenamep error.)
Quite unexpectedly, I also received an email with someone telling me that Emacs has yet another way to persist variables, and – surprise! – it’s built-in, too! Multisession variables are very similar to what persist offers, with one important distinction – you need to use the function multisession-value to access the value of a multisession variable. It also works as a setter, so (setf
(multisession-value my-variable) my-value) is the way to set or update a value of such a variable. (If you wonder how it’s possible that multisession-value is a function and not a macro, the answer is simple – multisession variables are in fact wrapped in objects holding more than just the current value of the variable, and multisession-value is made to be a generalized variable, something I might be tempted to write about in a future blog post.)
Other than the syntax differences, multisession variables work very similarly to persisted variables, with some (minor) differences. One of them is that they support running more than one Emacs session simultaneously, although this support is very rudimentary – there is no locking, so if two Emacs sessions try to update the same variable at the same time, the result is unpredictable. More interestingly, there are two backends for multisession variables – they can be kept in files within ~/.emacs.d (much like persist) or in an SQLite database (provided your Emacs was compiled with SQLite support, which it really should). Also, each multisession variable can contain information about the package it belongs to – this is useful when you say M-x list-multisession-values to see the list of all multisession variables, together with their packages and values, and the ability to interactively edit or delete them.
This means that multisession variables are a valid (and perhaps even better) alternative to persisted variables, slightly more expensive in terms of code complexity, but having a few features you might consider worth it (and built-in, although persist almost is, too, being on Elpa). And as is often the case, this means that you – as an Elisp programmer – have a choice which one to choose. (The main reason I use persist is that until very recently I had no idea about multisession variables. In fact, when I wrote my first code using persist, multisession variables didn’t even exist – they are a relatively new addition to Emacs, dating to December 2021, just a few weeks after I used persist for the first time.)
That’s it for today, happy hacking – and happy persisting your variables
!