![[Steve Friedl Logo]](/images/unixwiz-logo-140x80.gif){kind=logo}
This site uses advanced css techniques
Secure Shell is widely used to provide secure access to remote systems, and everybody who uses it is familiar with routine password access. This is the easiest to set up, is available by default, but suffers from a number of limitations. These include both security and usability issues, and we hope to cover them here.
In this paper, we'll present the various forms of authentication available to the Secure Shell user and contrast the security and usability tradeoffs of each. Then we'll add the extra functionality of agent key forwarding. We hope to make the case that using ssh public key access is a substantial win.
In the process, we'll be illustrating this with specific setup data
for VanDyke Software's SecureCRT 5.0, as well as notes for VShell.
SSH supports access with a username and password, and this is little more than an encrypted telnet. Access is, in fact, just like telnet, with the normal username/password exchange.
We'll note that this exchange, and all others in this paper, assume that an initial exchange of host keys has been completed successfully. Though an important part of session security, host validation is not material to the discussion of agent key forwarding.
All examples start from a user on homepc (perhaps a Windows workstation) connecting with SecureCRT to a server running OpenSSH. The particular details (program names, mainly) vary from implementation to implementation, but the underlying protocol has been proven to be highly interoperable.
| 1 | The user makes an initial TCP connection and sends a username. We'll note that unlike telnet, where the username is prompted as part of the connected data stream (with no semantic meaning understood by telnet itself), the username exchange is part of the ssh protocol itself. |  |
| 2 | The ssh daemon on the server responds with a demand for a password, and access to the system has not yet been granted in any way. |  |
| 3 | The ssh client prompts the user for a password, which is relayed through the encrypted connection to the server where it is compared against the local user base. |  |
| 4 | If the user's password matches the local credential, access to the system is granted and a two-way communications path is established, usually to a login shell. |  |
The main advantage of allowing password authentication is that it's simple to set up — usually the default — and is easy to understand. Systems which require access for many users from many varying locations often permit password auth simply to reduce the administrative burden and to maximize access.
| Pro: | easy to set up |
| Con: | allows brute-force password guessing |
| Con: | requires password entry every time |
The substantial downside is that by allowing a user to enter a password, it means anybody is allowed to enter a password. This opens the door to wholesale password guessing by users or bots alike, and this has been an increasingly common method of system compromise.
Unlike prior-generation ssh worms, which attempted just a few very common passwords with common usernames, modern badware has very extensive dictionaries of usernames and passwords and has proven to be most effective in penetrating even systems with "good" passwords. Only one compromised account is required to gain entry to a system.
But even putting security issues aside, the other downside of password authentication is that passwords must be remembered and entered separately upon every login. For users with just one system to access, this may not be such a burden, but users connecting to many systems throughout the day may find repeated password entry tedious.
And having to remember a different password for every system is not conducive to choosing strong passwords.
To counteract the shortcomings of password authentication, ssh supports public key access: the user creates a pair of public and private keys, installs the public key in his account on the target server, and access is granted on a much more secure (and more convenient) basis.
The keys are very long strings of numbers and are mathematically related, and access is granted by the server when it proves that it has the private key. In this protocol, the private key itself never travels over the wire, so there's nothing for an attacker to intercept. It's really very clever.
Both keys are stored in files, and the user's private key is further protected by a secret passphrase: when the private key is needed for authentication, the user must present this passphrase to unlock the key. This is an added protection so that a user's private key file is of no value if it falls into the wrong hands.
The public key, on the other hand, is truly public: it's not sensitive information and need not be protected. Keys are represented in ASCII as long strings of digits along with some accessory information such as the key type and a comment about the key (often the owner's name). There are a few formats for these keys:
---- BEGIN SSH2 PUBLIC KEY ---- Subject: steve Comment: "steve@unixwiz.net" AAAAB3NzaC1yc2EAAADAAQABAAAAgQCiR1Jo2hMbZp+XOIS6R+U9TrSxJjIMfk/SikL4 v38KINrPBUfmC2QxhhyP3zlthXRpzqJHbdp1wRodRiCr7Dx6JcxDQRh85IO0AzlKdQUi LwQ41Gz5WOtSGCbQ1Eag7EZaJ0XLdHvWYrNOo0mWcjYsuDlDgoeRSHWRsnrwGvMPv1== ---- END SSH2 PUBLIC KEY ----
ssh-rsa AzAAB3NzaC1yc2EaaaabiWaaaieaX9AyNR7xWnW0eI3x2NGXrJ4gkQpK/EqpkveGCvvbM \ oH84zqu3Us8jSaQD392JZAEAhGSoe0dWMBFm9Y41VGZYmncwkfTQPFH1P07vDw49aTAa2RJNFyV \ QANZCbSocDeuT0Q7usuUj/v8h27+PqsUUl9XVQSDIhXBkWV+bJawc1c= Steve's key
The OpenSSH server consults the keys in the file $HOME/.ssh/authorized_keys in the one-line format, and it grants access to the holder of the private key, all without a password. Note: older OpenSSH installations look in the file authorized_keys2 for version 2 keys, which is all we're considering here.
For VShell, the entire (small) public key file is placed into the per-user public key directory, and it can contain as many public keys as desired.
Transferring the public key files can be done in several ways, including cut-and-paste, file transfer, or emailing it to the system's administrator with a request to install the key.
But when SecureCRT is connected to a SSH server supporting the publickey-server subsystem (VShell, or OpenSSH with the publickey-server patch), it's even easier. The Public Key Assistant... (from the Tools menu) provides a mechanism to transfer a public key automatically. It could hardly be easier.
However the public key files are generated and placed on the remote system, it's a one-time operation: after that, the keys "just work", and now we'll look at the mechanism which takes place to provide this access.
| 1 | The user makes an initial connection and sends a username along with a request to use a key. |
|
| 2 | The ssh daemon on the server looks in the user's authorized_keys file, constructs a challenge based on the public key found there, and sends this challenge back to SecureCRT. |
|
| 3 | SecureCRT receives the key challenge. It finds the user's private key on the local system, but it's protected by an encrypting passphrase. An RSA key file is named id_rsa on OpenSSH and SecureCRT: other clients or keytypes use different filename formats. |
|
| 4 | SecureCRT prompts the user for the passphrase, and the comment on the private key is displayed to remind the user which key is requested. |
![[SecureCRT Passphrase Prompt]](../images/securecrt-key-prompt.gif)
|
| 5 | SecureCRT uses the private key to construct a key response, and sends it to the waiting sshd on the other end of the connection. It does not send the private key itself! |
|
| 6 | sshd validates the key response, and if valid, grants access to the system. |  |
This process involves more steps behind the scenes, but the user experience is mostly the same: you're prompted for a passphrase rather than a password. But, unlike setting up access to multiple computer systems (each of which may have a different password), using public key access means you type the same passphrase no matter which system you're connecting to.
| Pro: | public keys cannot be easily brute-forced |
| Pro: | the same private key (with passphrase) can be used to access multiple systems: no need to remember many passwords |
| Con: | requires one-time setup of public key on target system |
| Con: | requires unlocking private key with secret passphrase upon each connection |
This has a substantial, but non-obvious, security benefit: since you're now responsible for just one secret phrase instead of many passwords, you type it more often. This makes you more likely to remember it, and therefore pick a stronger passphrase to protect the private key than you otherwise might.
Trying to remember many separate passwords for different remote systems is difficult, and does not lend itself to picking strong ones. Public key access solves this problem entirely.
We'll note that though public-key accounts can't generally be cracked remotely, the mere installation of a public key on a target system does not disable the use of passwords system wide. Instead, the server must be explicitly configured to allow only public key encryption by the use of the PasswordAuthentication no keywords in the sshd_config file.
Now that we've taken the leap into public key access, we'll take the next step to enable agent support. In the previous section, the user's private key was unlocked at every connection request: this is not functionally different from typing a password, and though it's the same passphrase every time (which makes it easier to remember), it nevertheless gets tedious in the same way.
Fortunately, the ssh suite provides a broker known as a "key agent" which can hold and manage private keys on your workstations, and responding to requests from remote systems to verify your keys. Agents provide a tremendous productivity benefit, because once you've unlocked your private key (one time when you launch the agent), subsequent access works with the agent without prompting.
This works much like the key access seen previously, but with a twist.
| 1 | The user makes an initial connection and sends a username along with a request to use a key. |
|
| 2 | The ssh daemon on the server looks [1] in the user's authorized_keys file, constructs a challenge based on the key, and sends it [2] back to SecureCRT. |
|
| 3 | SecureCRT receives the key challenge, and forwards it to the waiting agent. The agent, though part of the SecureCRT process, is nevertheless a separate entity. The agent opens the user's private key and discovers that it's protected by a passphrase. |
|
| 4 | SecureCRT prompts the user for the passphrase to unlock the private key, just as with standalone key support. |
|
| 5 | The agent constructs the key response and hands it back [1] to SecureCRT, which sends it off [2] to the sshd waiting on the other end. Unlike the previous example, SecureCRT never sees the private key directly, only the key response. |
|
| 6 | sshd validates the key response, and if valid, grants access to the system. Note: the agent still retains the private keys in memory, though it's not participating in the ongoing conversation. |  |
| Pro: | Requires unlocking of the private key only once |
| Pro: | Facilitates scripted remote operation to multiple systems |
| Con: | One-time cost to set up the agent, though SecureCRT makes this very easy |
| Con: | Requires private key on remote client machines if they're to make further outbound connections |
As far as the user is concerned, this first exchange is little different from key access shown in the previous section: the only difference is which program prompts for the private key (the ssh client itself versus the agent). With SecureCRT, the passphrase prompts are identical.
But where agent support shines is at the next connection request made while the agent is still resident. Since it remembers the private keys from the first time it was unlocked with the passphrase, it's able to respond to the key challenge immediately without prompting. The user sees an immediate, direct login without having to type anything.
Many users only unlock their private keys once in the morning when they launch their ssh client and agent, and they don't have to enter it again for the rest of the day because the resident agent is handling all the key challenges. It's wonderfully convenient, as well as secure.
It's very important to understand that private keys never leave the agent: instead, the clients ask the agent to perform a computation based on the key, and it's done in a way which allows the agent to prove that it has the private key without having to divulge the key itself. We discuss the challenge/response in a later section.
Once agent support is enabled, all prompting has now been bypassed, and one can consider performing scripted updates of remote systems. This contrived example copies a .bashrc login config file to each remote system, then checks for how much disk space is used (via the df command):
# scripted update of several remote systems for svr in server1 server2 server3 server4 do scp .bashrc $svr:~/ # copy up new .bashrc ssh $svr df # ask about disk space done
Without agent support, each server would require two prompts (first for the copy, then for the remote command execution). With agent support, there is no prompting at all.
However, these benefits only accrue to outbound connections made from the local system to ssh servers elsewhere: once logged into a remote server, connecting from there to yet a third server requires either password access, or setting up the user's private key on the intermediate system to pass to the third.
Having agent support on the local system is certainly an improvement, but many of us working remotely often must copy files from one remote system to another. Without installing and initializing an agent on the first remote system, the scp operation will require a password or passphrase every time. In a sense, this just pushes the tedium back one link down the ssh chain.
Fortunately, there's a great solution which solves all these issues.
With our Key Agent in place, it's time to enable the final piece of our puzzle: agent forwarding. In short, this allows a chain of ssh connections to forward key challenges back to the original agent, obviating the need for passwords or private keys on any of the intermediate machines.
| 1 | This all starts with an already established connection to the first server, with the agent now holding the user's private key. The second server plays no part yet. |
|
| 2 | The user launches the ssh command-line client on the first server with a request to connect to server2, and this passes the username and a use-key request to the ssh daemon (this could likewise be done with the command-line clients as well). |
|
| 3 | The ssh daemon consults the user's authorized_keys file [1], constructs a key challenge from the key, and sends it [2] back down the channel to the client which made the request. |
|
| 4 |
This is where the magic occurs: the ssh client on server receives the key challenge from the target system, and it forwards [1] that challenge to the sshd server on the same machine acting as a key agent. sshd in turn relays [2] the key challenge down the first connection to the original SecureCRT. Once back on homepc, SecureCRT takes the final step in the relay process by handing the key challenge off [3] to the resident agent, which knows about the user's private key. |
|
| 5 |
The agent running on homepc constructs the key response and hands it [1] back to SecureCRT, which in turn passes it [2] down the channel to the sshd running on server. Since sshd is acting as a key agent, it forwards [3] the key response off to the requesting ssh client, which sends it [4] to the waiting sshd on the target system (server2). This forwarding action is all done automatically and near instantly. |
|
| 6 | The ssh daemon on server validates the key response, and if valid, grants access to the system. |
|
This process can be repeated with even more links in the chain (say, if the user wanted to ssh from server2 to server3), and it all happens automatically. It supports the full suite of ssh-related programs, such as ssh, scp (secure copy), and sftp (secure FTP-like file transfer). SecureCRT's command-line utilities (vcp, vsftp, and vsh) participate as well.
| Pro: | Exceptional convenience |
| Con: | Requires installation of public keys on all target systems |
| Con: | Requires a Tech Tip to understand |
| Pro: | An excellent Tech Tip is available  |
This does require the one-time installation of the user's public — not private! — keys on all the target machines, but this setup cost is rapidly recouped by the added productivity provided. Those who use public keys with agent forwarding rarely go back.
One of the difficulties with forbidding password authentication is installation of public keys for new users: since they cannot login without a key, we're in a chicken-and-egg situation with respect to their own ability to set up their own accounts. This usually means that the administrator has to do it, and this is two interruptions for each new user rather than just one. This doesn't scale so well for larger enterprises.
VShell provides a clever solution to this: even with password authentication disabled, it can be configured to permit a small number — usually three — of password attempts before it reverts to key-only mode. This gives the user a couple of tries to login with a password to install a public key before the key is the only way to get in.
Password attempts are maintained on a per-user basis, so allowing the new users to make a go of keyless logins doesn't open this door for the existing accounts.
Configuring this requires coordination of a few settings, but it's easy to do:
It's inevitable that some users will have problems logging in even within the allotted attempts, and this will certainly require administrative intervention, but for the majority of users it means self-service installation of public keys.
One of the more clever aspects of the agent is how it can verify a user's identity (or more precisely, possession of a private key) without revealing that private key to anybody. This, like so many other things in modern secure communications, uses public key encryption.
When a user wishes access to an ssh server, he presents his username to the server with a request to set up a key session. This username helps locate the list of public keys allowed access to that server: typically it's found in the $HOME/.ssh/authorized_keys file.
The server creates a "challenge" which can only be answered by one in possession of the corresponding private key: it creates and remembers a large random number, then encrypts it with the user's public key. This creates a buffer of binary data which is sent to the user requesting access. To anybody without the private key, it's just a pile of bits.
When the agent receives the challenge, it decrypts it with the private key. If this key is the "other half" of the public key on the server, the decryption will be successful, revealing the original random number generated by the server. Only the holder of the private key could ever extract this random number, so this constitutes proof that the user is the holder of the private key.
The agent takes this random number, appends the SSH session ID (which varies from connection to connection), and creates an MD5 hash value of the resultant string: this result is sent back to the server as the key response.
The server computes the same MD5 hash (random number + session ID) and compares it with the key response from the agent: if they match, the user must have been in possession of the private key, and access is granted. If not, the next key in the list (of any) is tried in succession until a valid key is found, or no more authorized keys are available. At that point, access is denied.
Curiously, the actual random number is never exposed in the client/agent exchange - it's sent encrypted to the agent, and included in an MD5 hash from the agent. It's likely that this is a security precaution designed to make it harder to characterize the properties of the random number generator on the server by looking at the client/agent exchange.
More information on MD5 hashes can be found in An Illustrated Guide to Cryptographic Hashes.
One of the security benefits of agent forwarding is that the user's private key never appears on remote systems or on the wire, even in encrypted form. But the same agent protocol which shields the private key may nevertheless expose a different vulnerability: agent hijacking.
Each ssh implementation has to provide a mechanism for clients to request agent services, and on UNIX/Linux this is typically done with a UNIX domain socket stored under the /tmp/ directory. On our Linux system running OpenSSH, for instance, we find the file /tmp/ssh-CXkd6094/agent.6094 associated with the SSH daemon servicing a SecureCRT remote client.
This socket file is as heavily protected as the operating system allows (restricted to just the user running the process, kept in a protected subdirectory), but nothing can really prevent a root user from accessing any file anywhere.
If a root user is able to convince his ssh client to use another user's agent, root can impersonate that user on any remote system which authorizes the victim user's public key. Of course, root can do this on the local system as well, but he can do this directly anyway without having to resort to ssh tricks.
Several environment variables are used to point a client to an agent, but only SSH_AUTH_SOCK is required in order to use agent forwarding. Setting this variable to a victim's agent socket allows full use of that socket if the underlying file is readable. For root, it always is.
# ls -l /tmp/ssh* — look for somebody's agent socket /tmp/ssh-CXkd6094: total 24 srwxr-xr-x 1 steve steve 0 Aug 30 08:46 agent.6094= # export SSH_AUTH_SOCK=/tmp/ssh-CXkd6094/agent.6094 # ssh steve@remotesystem remote$ — Gotcha! Logged in as "steve" user on remote system!
One cannot tell just from looking at the socket information which remote systems will accept the user's key, but it doesn't take too much detective work to track it down. Running the ps command periodically on the local system may show the user running ssh remotesystem, and the netstat command may well point to the user's home base.
Furthermore, the user's $HOME/.ssh/known_hosts file contains a list of machines which the user has a connection: though they may not all be configured to trust the user's key, it's certainly a great place to start looking. Modern versions (4.0 and later) of OpenSSH can optionally hash the known_hosts file to forestall this.
There is no technical method which will prevent a root user from hijacking an SSH agent socket if he has the ability to access it, so this suggests that agent forwarding might not be such a good idea when the remote system cannot be entirely trusted. All ssh clients provide a method to disable agent forwarding.
Most users spend the bulk of their time in SecureCRT terminal windows, but there's a parallel set of tools to support command-line usage. Available from VanDyke Software, they fully work and play well with all the other parts, including the key agents.
C:\Temp> vcp file*.jpg steve@unixwiz.net:/home/www/htdocs/images Processing directory: . ls: .\file1.jpg ls: .\file2.jpg ls: .\file3.jpg Copying: .\file3.jpg 100% 57KB 57KB/s 00:00:01 Copying: .\file2.jpg 100% 327KB 40KB/s 00:00:08 Copying: .\file1.jpg 62% 216KB 43KB/s 00:00:02 ETA — still in progress; shows estimated time
Each takes a similar set of command-line parameters which direct most of the important behaviors, and this makes them all highly amenable to scripting.
Of particular utility in the scripting arena is the ability to provide a specific private key file: it's common to create a separate key which provides only limited powers on the target server (for instance, it could allow the user to connect, but only to run a single, specified command via vsh).
Since scripts often run from Scheduled Tasks (and do not allow user interaction), all the credentials necessary for remote access mdash; they username, the key, and the passphrase — must be provided with the script. This is not terribly secure, so by creating a key configured with purpose-built limited powers, compromise of the credentials by those with access to the machine doesn't provide unrestricted access to the remote system.
Up to this point, we've provided essentially no practical how-to information on how to install or configure any particular SSH implementation. Our feeling is that this information is covered better elsewhere, and we're happy to provide some links here to those other resources.
The Secure Shell: The Definitive Guide, 2 Ed
(O'Reilly & Associates).
This Tech Tip is unpublished.
![[Steve's Email]](/images/steve-email.gif)
![[RSS Feed available]](/images/feed-icon-14x14.png){kind=icon}