GPG agent forwarding via SSH

03 Mar 2021

Have you ever wanted to combine GPG, GPG agent, and SSH socket forwarding? It’s something I’ve been looking forward to for a while. The last time I tried it, many years ago, it just wasn’t ready. But a few days ago I tried it out again and it all works! Read on for an explanation of what these things are, and why they’re useful.

What Are All These Things?

GPG

GPG (GNU Privacy Guard) is an application that lets you encrypt/decrypt files. For example, I might want to keep some API key handy, but not leave it lying around in plain-text. Instead of simply storing the API key in a plain-text file, I’ll encrypt the API key using GPG, then save the encrypted result in a file. Whenever I need to use the API key, I’ll temporarily decrypt it using GPG, and pass it to whatever program needs the key, e.g. through a pipe, without ever saving the plain-text API key to disk.

GPG Agent

GPG uses keys to encrypt/decrypt data, and these need to be available while running GPG. If we simply stored the GPG keys on disk in plain-text, anyone who gained access to the filesystem could easily use the GPG keys to decrypt any data that GPG had created. Luckily, GPG supports encrypting keys with a pass-phrase, and storing the keys in an encrypted form, which prevents trivial theft of the keys.

(Side-note: Use full disk encryption too. GPG key pass-phrases also mitigates attacks by malicious code that somehow executes as your own user ID.)

However, this makes repetitive use of GPG a bit annoying; you have to type in your (preferably long and secure) GPG key pass-phrase every time you use it. To aid ease-of-use, you can run a program called a GPG agent. The GPG agent manages loading and decrypting of keys. Once a key is used, the GPG agent caches it in (locked, non-swappable) memory for a short time. If the key is used again while it’s cached, there’s no need to re-enter the pass-phrase.

How does this work? The GPG agent is a long-running background program. It listens on a Unix domain socket for connections from the user-run GPG application. Each time GPG runs, it connects to this domain socket to acquire keys.

How is this secure? The Unix domain socket can only be accessed by processes running on the same machine as the GPG agent. Unix domain sockets can only be accessed via the filesystem or other local means. File system permissions protect the domain socket against access by most other local users, with the exception at least of root.

SSH

SSH (Secure Shell) is a utility that enables secure command-line access to remote machines. The most basic usage is to run the SSH client application on your local system, and connect to a remote system, yielding the ability to execute shell commands remotely. However, SSH has many other features.

One of these features is the ability to forward sockets/ports between the local and remote system. This can be used in many ways, such as to allow a network client application running on the remote system to connect to a network server application running on (or accessible from) the local system, whilst tunnelling all the traffic inside the secure and encrypted SSH session.

Historically, SSH only supported forwarding TCP sockets, but in more recent versions, support was added for forwarding Unix domain sockets too.

Combining Them Together, and Why?

So: GPG agent listens on a Unix domain socket. SSH now supports forwarding Unix domain sockets. Can we combine the two so that we can SSH to a remote system, run GPG there, and have it connect to a local GPG agent? Yes, we can!

What’s the use-case? I’m working from home, as many people are these days. I need to run a script on my desktop in the office. The script needs to make requests to a web-service, and needs an API token to do so. As mentioned before, the API token is stored in a GPG-encrypted file on my office desktop. Without GPG agent forwarding, every time I run the script, I need to type my GPG key pass-phrase into the SSH session, which is repetitive and error-prone. I could run a GPG agent on the desktop, but then I’d have to manage starting and stopping the GPG agent on the desktop, and still have to unlock the key every time I connected and started the agent. I would need to repeat this process for each remote machine that I access. If I could forward the GPG agent connection from my work desktop to my laptop, I could manage unlocking GPG keys in one place; on the laptop.

A Security Warning

There are some disadvantages (but also advantages) to SSH and GPG agent forwarding. Please familiarize yourself with them before deciding to use these features.

Disadvantages

Any process on the remote machine that can access the forwarded socket can make use of your keys. If the agents protocol used a TCP network socket, this would be a critical flaw, since TCP sockets are quite widely accessible in general by default; even restricting access to localhost is nowhere near enough to be secure. However, the agent protocols run over Unix domain sockets, which allow for access control. However, Unix domain sockets are still subject to undesired access, e.g. by root, or if socket file permissions are accidentally set permissively. In general, you should consider that any user of a machine to which you connect with agent forwarding enabled may be able to make use of your keys. This also applies to any unexpected/unknown or malicious software that’s running under your uid.

In my use-case, I’m connecting to machines where I’m the only user, so I have accepted the risk. This condition may not apply in your case!

Advantages Too!

Without agent forwarding, I would need to copy my (pass-phrase-protected) SSH and GPG private keys to any remote system where I needed to use them. This increases the attack surface; the more copies there are, the higher the possibility of the keys being compromised. However, when using agent forwarding, the private (and security-critical) portions of the keys don’t need to be present on machines where the keys are used; the agent performs all the key-related operations locally. This increases security since fewer copies of private keys are required.

Note: Since I only recently set up GPG agent forwarding, I haven’t actually experimented with removing my GPG private keys from the remote machines where I’m making use of agent forwarding. I hope to try this out soon.

Another nice feature of some SSH (and GPG?) agents is that they can alert you when an already-unlocked key is used. The alert will be accompanied by a yes/no prompt so that you can deny access. This is much easier than re-entering a pass-phrase, but still allows for confirmation of key use. This worked well for me when I was running an older version of Ubuntu on my laptop, which contained a different SSH agent. Unfortunately the latest desktop-oriented SSH agents don’t seem to implement this feature:-( I’m not sure whether any GPG agent(s) implement it either. This would be a nice feature for somebody to resurrect.

Finally, at least with SSH agent forwarding, when you disconnect from a machine, you may be certain that access-to/use-of keys is not possible by that machine, since the keys themselves are only ever accessible on the client. Note: With SSH agent forwarding, I believe that private key operations are performed by the agent, and results returned to the client, rather than the private key itself. I imagine that the GPG agent protocol works in a similar way. However, I am not sure on this point.

Version Notes

The rest of this post is written based on forwarding a GPG agent running on an Ubuntu 20.04 system to other systems. My system runs the XFCE desktop environment, which may influence the selection or operation of the GPG agent. You may find differences in behavior and paths if you’re running a different Linux distribution, different SSH or GPG versions, or a different desktop environment.

Server Setup

You will almost certainly want to enable the following option in /etc/ssh/sshd_config on each remote system:

StreamLocalBindUnlink yes

Whenever a Unix domain socket is forwarded, SSH needs to create the socket in some filesystem path on the remote system. If the path already exists, this will fail. This option tells the SSH server to delete the path before creating the socket, to maximize the chance of success.

You will need to restart the SSH daemon after making this change.

GPG Agent Paths

How do we know which Unix domain socket to forward? Older versions of GPG stored the socket path in environment variable GPG_AGENT_INFO. However, this variable is now obsolete. Instead, GPG seems to rely on some well-known pathname these days. This may be related to whether various desktop/systemd integration features are enabled. The simplest way to find the path is to simply run gpg under strace on each system (local and remote), and see which socket path it connects to. Then, we can set up a forwarding between these two paths.

strace -e connect gpg -d /path/to/file.gpg
connect(6, {sa_family=AF_UNIX, sun_path="/run/user/1000/gnupg/S.gpg-agent"}, 34) = 0

In my case, I find the following paths:

/run/user/1000/gnupg/S.gpg-agent on my Ubuntu 20.04 laptop, where I am logged in as uid 1000; a typical value for the first user added to an Ubuntu system.
/run/user/4321/gnupg/S.gpg-agent on a system I connect to where I’ve chosen my uid to match an externally-managed user database.
/home/swarren/.gnupg/S.gpg-agent on an older system that I connect to, and where for whatever reason GPG doesn’t connect to an agent socket in /run.

`S.gpg-agent.extra`

(At least the version I’m running of) GPG agent actually listens on a number of paths. S.gpg-agent is intended for local use. S.gpg-agent.extra is apparently a more restricted channel, and more suitable for use with agent forwarding. You probably want to forward this path.

Manual Forwarding

Before trying agent forwarding, you probably want to ensure that you can simply SSH to the relevant remote host and run GPG there without issue. You’ll need your keys on the remote host and to either manually enter your GPG key pass-phrase or have a agent running on the remote machine. Once this is working, you will have debugged any issues solely related to GPG itself, separately from the added complication of using GPG agent forwarding.

SSH port forwarding is available from the command-line:

ssh -R /home/swarren/.gnupg/S.gpg-agent:/run/user/1000/gnupg/S.gpg-agent.extra somehost
gpg -d /path/to/file.gpg
# Now, your local agent should provide the key,
# or prompt you for the key pass-phrase.

That’s a bit of a mouthful to type. Luckily, we can automate it!

Automated Forwarding

We can add entries to ~/.ssh/config to request SSH to enable forwarding without having to specify options on the command-line each time. For example:

Host somehost
    RemoteForward /home/swarren/.gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra

Conditional Forwarding

You may need to use different socket paths for different pairs of machines:

Host somehost
    RemoteForward /home/swarren/.gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra 
Host otherhost
    RemoteForward /run/user/4321/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra

You may need to vary your configuration based on which system you’re running SSH from. For example, so as to only forward from some machines, or to forward a different agent path from each machine. You can use ssh_config’s Match rules to do this:

Match host somehost exec "[ $(hostname) == laptop ]"
    RemoteForward /home/swarren/.gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra 
Match host somehost exec "[ $(hostname) == desktop ]"
    RemoteForward /home/swarren/.gnupg/S.gpg-agent /run/user/9876/gnupg/S.gpg-agent.extra

Finally, you might not want to enable agent forwarding all the time. Perhaps you want to ssh somehost normally, but be able to ssh somehost-agent when you need agent forwarding. That would limit the amount of time your agent socket is forwarded and exposed to code running on the remote system. You can combine the previous techniques, plus ssh_config’s Hostname statement to create a virtual destination hostname that works this way:

Match host somehost-agent exec "[ $(hostname) == laptop ]"
    Hostname somehost.domain
    HostKeyAlias somehost.domain
    RemoteForward /home/swarren/.gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra

(I haven’t actually tried out that last option yet, but it’s a combination of configuration options that I have already used separately.)

Gotchas and Negative Interactions

If you mix local interactive and SSH logins to a system, you may find that GPG agent forwarding breaks use of the GPG agent from local sessions. That’s because GPG agent forwarding deletes and replaces any agent socket that might have been provided by a local GPG agent. When switching back to a local interactive login, you may need to restart the daemon. If you log out and log in, rather that simply locking your screen, this probably happens automatically. Otherwise, you might need to run a command such as:

systemctl --user restart gpg-agent

When using GPG agent forwarding, it’s possible you need to stop any GPG agent running on the remote system:

systemctl --user stop gpg-agent

… then log out of SSH and reconnect. While experimenting, I believe stopping the GPG agent on the remote machine helped bypass some issues, but I may simply be imagining this, and have conflated the issue with having accidentally specified the wrong agent paths to forward.