Does this site look plain?

This site uses advanced css techniques

[OpenSSH logo] With the Core-SDI Advisory on the SSH1 CRC-32 compensation attack detector vulnerability and the associated exploit running around in Fall 2001, many people are rushing to upgrade their vulnerable secure shell servers. The exploit appears to be easy to use and quite effective, rooting boxes right and left.

Table of Contents
  1. Locate the tarballs
  2. Building everything
  3. Creating OpenSSH public keys
  4. Configuring incoming SSH public key access
  5. Using VanDyke SecureCRT with OpenSSH
  6. Post-public key configuration
  7. Copying the binaries elsewhere
  8. Manual host key generation
  9. References
  10. Upgrading for the ZLIB problem
  11. Portability notes
  12. Privilege Separation
  13. Related Resources

We were among those rushing around upgrading systems, and it was quite a chore. So we've written down some rough instructions that report how we did it on our own systems. These are not intended to be tutorial, but they might give somebody a start. These instructions only apply to UNIX systems, and in particular we have installed on Red Hat Linux 6.2 and Solaris 2.7.

And we wish to note that this tip was created years ago, and much has changed since then even though the fundamentals should be familiar. Always check your local documentation to see what's different in the version you're on now.


Locate the tarballs

There are three separate packages that have to be installed, and each is found in a different place throughout the World Wide Web.

zlib
This is a great file compression library, and from the official home page the mirrors can be located. We used version 1.1.4 in tarball zlib-1.1.4.tar.gz. Note that there is a substantial security issue in version 1.1.3 - upgrade and rebuild your server if you've got the older one.
OpenSSL
This is a secure socket library, and it is easy to find at the OpenSSL source code specifically do not need the "engine" variant, which uses hardware that does the heavy-duty crypto. We've noted that this web site is often unavailable.
OpenSSH
This is the SSH client & server itself, and it's really easy to get the wrong version. We found ours at ftp://ftp3.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/, and it's important to always look in the portable/ subdirectory - otherwise you'll get a BSD only version. Our tarball was openssh-3.0p1.tar.gz.

We typically do our work in a /home/source area, so the unpacking will look like this:

#  cd /home/source 
#  gtar -xzvf zlib-1.1.4.tar.gz 
#  gtar -xzvf openssl-0.9.7a.tar.gz 
#  gtar -xzvf openssh-3.5p1.tar.gz 

Building everything

The three packages have to be built in order, and we do them each in turn:

#  cd zlib-1.1.4 
#  ./configure 
#  make 
#  make install 

#  cd ../openssl-XXXXX 
#  ./config                         Note: not './Configure'
#  make 
#  make test 
#  make install 

#  cd ../openssh-XXXXX 
#  ./configure --with-ipv4-default --with-md5-passwords 
#  make 

#  no "make install" yet! 

We chose the arguments to ./configure by feeling around, and we think this is the right set for our environment. These instructions have been working for us on a number of systems.

--with-ipv4-default appears to tell SSH to use only IPv4 -- "regular" internet addresses -- instead of the next-generation IPv6 addresses. We're not sure if this is required or not.

--with-md5-passwords enables the better password hashing as found in Linux, and it probably isn't required on other systems or if you only use RSA authentication.

We suspect that --with-pam might also be useful, but so far the system is working fine for us without it.

NOTE - for recent releases of OpenSSH, this is where the sshd user and group should be added to support privilege separation. See the end of this document for the details.

Install Everything - First Time

If this is a first-time installation of the Secure Shell on this system, it's safe to just do a make install in the OpsnSSH area and it will drop its files in the proper places.

But if there is a working SSH installation then it's wise to go find the key setup files and save them off to the side first. This installation also parks files in different target directories than the older one we used, so it's wise to poke around and save or delete the old stuff. We typically look in these directories:

/etc
/usr/local/bin
/usr/local/sbin

Pay particular note to the ssh_config and sshd_config files, which define important server and client options. Save these files to the side.

It's also wise to keep an alternate method of accessing the system, such as the console or telnet, until you're really sure that you have gotten SSH installed properly. It's a drag to find (for instance) that sshd doesn't start properly on a reboot when you have no keyboard or monitor attached to the server under the desk. Been there, done that.

Once these precautions have been taken, do a make install in the SSH source directory, and it will put everything where it belongs.

Post-Install Configuration

Once SSH is installed, there are still a few configuration steps required. We're really not terribly strong with this, and in any case cannot provide a tutorial for SSH here. But we will note a few things that ought to be attended to after upgrading secure shell.

Setting up Public Key support

We are not excited about allowing PasswordAuthentication for our secure shell daemons, because this provides limited system protection. It's true that it does forestall network sniffing -- and this is no small matter -- but it does not stop the bad guy from running a password guesser over the secure channel. In our work as penetration testers, we've gotten into systems via secure shell this way: by guessing or finding a password.

Much better is to use Public Key encryption, which has the potential for substantially better security as well as more convenience. We've been surprised how much we like public key support in SSH - much more than we expected.

In PK, each user creates a pair of keys: one public and one private, and the two are mathematically related. Both are very long strings of binary numbers, and they bear no relation to anything like a "password" or anything else human readable. The public key can be given away, but the private key is so important that it is further protected by a "pass phrase". This pass phrase actually encrypts the private key, and you'll need to type this key every time you need to use it.

The pass phrase should be something easy to type but impossible to guess: you'll be typing it a lot. At first it seems annoying to type this same phrase over and over, but quickly found that typing one long phrase for everything positively beats the pants off having to remember many individual passwords for all the sites we visit. This has turned out to be a big win.

Public key support revolves around the .ssh/ subdirectory in each user's UNIX/Linux home directory, and a couple of important files live here. These include (but not an exhaustive list):

IMPORTANT - Nothing will work right unless the ownership and permissions are set correctly. The directory and all files must be owned by the user in question, and must be unreadable by anybody but the user. The ssh server will simply ignore files that don't meet this stringent rule - it's a good security precaution. Just remember to chmod og= ~/.ssh ~/.ssh/* every now and then to be sure.

Creating OpenSSH public keys

The ssh-keygen program is used to create a public key pair suitable for OpenSSH client and server use (there are other mechanisms for creating keys, we'll touch on the m later). It's possible to create both "user" and "host" keys, but since the host keys are usually created during the software installation process, we'll only touch on the user key.

There are many options for creating public key pairs, but we only care about making SSH version 2 RSA keys. This command creates the key pair, prompts for a pass phrase to encrypt the private key, and stores the two files in your ~/.ssh directory:

$ ssh-keygen -b 1024 -t rsa

-b 1024 specifies the number of bits to encrypt the key, and the consensus is that 1024 is fine: larger values just slow things down without adding appreciably to security. -t rsa specifies to use RSA encryption to produce an SSH version 2 key (the default is to create a version one key - bad news). It prompts for a filename to store the keys in (usually OK to keep the default), then asks for a pass phrase twice.

Choose a good pass phrase! Your private key is not very private if the very secret id_rsa file can be compromised.

Running the command should look like this:

$ ssh-keygen -b 1024 -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/friedl/.ssh/id_rsa): RETURN
Enter passphrase (empty for no passphrase): passphrase + RETURN
Enter same passphrase again: passphrase + RETURN
Your identification has been saved in /home/steve/.ssh/id_rsa.
Your public key has been saved in /home/steve/.ssh/id_rsa.pub.
The key fingerprint is:
67:2c:06:ff:5a:cf:36:64:11:20:74:90:9f:97:14:8e steve@unixwiz.net

Now the files ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa contain your public and private keys, respectively. We'll use them shortly. The public key file is one line of ASCII, and it looks like this (though it's actually all one line):

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAx9aYnr7XwNw0Ei3X2ngxRjGg
kQpK/EqpkveGCvvbMoH84zqu3Us8jSC0D392JZAEohGSoe0dWMBFm9Y41VGZ
YmncwkfTQPFH1P0IvDw49aTAa2RJNFyVQANZCbSocDeTT0QpusuUj/v8hj7+
PqsUUpUXVQSDIhXBkWV+bJawc1c= steve@unixwiz.net

Again, there is no need to keep a public key secret.

Configuring incoming SSH public key access

When the ssh server is considering an incoming connection request via the public key mechanisms, it consults the ~/.ssh/authorized_keys for version-2 keys (we're specifically ignoring version-1 key support).

To enable a remote user to the local OpenSSH server, simply edit the user's ~/.ssh/authorized_keys file and append the user's public key -- one line -- to the file and save it. Do make sure that the file is owned by the user and is not readable by anybody else, but there is no need to restart the OpenSSH server to make it take effect.

Then, from the remote system, connect with the ssh command and the name of the remote host.

$ ssh www.example.com
The authenticity of host 'www.example.com (10.1.1.1)' can't be established.
RSA key fingerprint is 50:29:ac:84:60:42:31:f7:e0:4c:bc:d2:ff:de:ba:89.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www.example.com,10.1.1.1' (RSA) to the list of known hosts.
Enter passphrase for key '/home/steve/.ssh/id_rsa': passphrase + RETURN
$ uname -a
Linux www.example.com 2.2.12-20 #1 Mon Sep 27 10:25:54 EDT 1999 i586 unknown
$

The first time you contact any host, you'll be warned that your local SSH installation doesn't know about the remote. When you confirm that you want to connect to it with yes, the name and fingerprint of the remote host is added to the ~/.ssh/known_hosts file. Then you won't be prompted again for this host.

After confirming the host, you'll enter your passphrase to decrypt your private key. If done correctly, the SSH client and server exchange your key information, and if all is well you're granted access to the remote system.

There is no reason that a single public key can't be put in multiple user accounts on the remote system: it's possible to store your public key in both /home/steve/.ssh and /root/.ssh directories. So logged in as a regular user, we can login as root on the remote system:

$ ssh -l root www.example.com

Using VanDyke SecureCRT with OpenSSH

We use the fantastic telnet/serial/SSH client SecureCRT (from VanDyke Software) to connect from our Windows (ME/NT/2000) boxes to OpenSSH, and setup is straightforward.

After SecureCRT is installed on the Windows box, a pair of public/private keys must be created. From the menu, select Options:Global Options, and from this dialog box select SSH2, then click Create Identity File. Key creation is done just one time after installation. NOTE: these instructions are for SecureCRT v4.x.

[Secure CRT]

Click Next in the Key Generation Wizard, then select an RSA - not DSA - key and click Next. In the next dialog box, enter your pass phrase (twice), then a comment that appears in the file. By convention this is your email address, but it can be any descriptive text that helps you identify the key. Then click Next.

In the next dialog box, select 1024 bits for the key and click Next. Then you are invited to move the mouse around: this provides important random inputs that help generate a key pair that will be hard to crack. This takes perhaps 30 seconds, and when finished you are asked to click Next.

In the final dialog box, you're asked to save your key files somewhere. It's not as important exactly where they go as long as you remember where that place is. You will need the public key file shortly. For our example we are using C:\identity.pub.

After the key has been generated, locate the public key file and get it over to the UNIX system. Since this file is small and ASCII, it's often easiest to open it in notepad, select the whole thing, and "cut" it into the Windows notepad. Then, logged into the remote server (perhaps with telnet or via ssh using a password), and pasting it into a file. For instance:

$ cat > /tmp/identity.pub
"paste" the file here
^D              <-- control-D
$

Remember that the entry is one line, so you might need to use your favorite editor and join them together if pasting created multiple lines.

Once the file is over on the UNIX system, it has to be converted from the SecureCRT format (some kind of standard) to that used by OpenSSH. The ssh-keygen command is able to do this conversion for you:

$ ssh-keygen -i -f /tmp/identity.pub >> ~/.ssh/authorized_keys

Now your SecureCRT public key is installed, so you should be able to ssh into your UNIX box from Windows.

Post-public key configuration

Once the public key mechanisms are known to be working correctly, including testing of proper restart after reboot, we recommend some additional steps.

Disable telnetd
Even with TCP wrappers limiting the IP addresses of incoming connection, we believe that telnet is just too dangerous to leave running. So we disable telnetd in the /etc/inetd.conf file by commenting out this line
#telnet stream  tcp     nowait  root    /usr/sbin/tcpd  in.telnetd
and "kicking" inetd by sending a hangup signal to it:
# ps -fCinetd
UID        PID  PPID  C STIME TTY          TIME CMD
root       413     1  0 Feb01 ?        00:00:00 [inetd]

# kill -1 413
We believe that newer systems use the xinetd.conf file, but we don't have this on any of our systems: the idea ought to be similar.
Now telnet won't accept connections any longer. The brave and determined might actually remove the telnet daemon altogether. We've done this on most of our systems.
Disable PasswordAuthentication option
... in the OpenSSH configuration file (often /usr/local/etc/sshd_config), find the setting for PasswordAuthentication and change the value to no. This then permits only public key authentication and prevents "regular" passwords from working. We feel strongly that allowing people to guess passwords is a really bad idea, and by insisting on RSA keys, a whole raft of shenanigans can be avoided.
Disable ChallengeResponseAuthentication option
Unless you use S/Key authentication (we never have), disable this — it's been the source of security troubles in the past.
Disable UsePAM option
PAM is the Pluggable Authentication Module mechanism on Linux, and it should be disabled if you wish to use key-only logins.

We're quite sure that other options in the config file are important, but we've not really looked at them in much detail. Shame on us.

Copying the binaries elsewhere

Once OpenSSH has been installed on a platform, it's reasonable to want to install it elsewhere without having to actually build it. So we've noted the files that are installed by the make install process so that they can be copied (with permissions!) to some other system. This listing is relative to /usr/local.

$ gtar -tzvf OpenSSH-3.1p1z.tgz
-rws--x--x root/root    211396 2002-03-13 01:32:57 bin/ssh
-rwxr-xr-x root/root     26520 2002-03-13 01:32:57 bin/scp
-rwxr-xr-x root/root     75564 2002-03-13 01:32:57 bin/ssh-add
-rwxr-xr-x root/root     42428 2002-03-13 01:32:57 bin/ssh-agent
-rwxr-xr-x root/root     79640 2002-03-13 01:32:57 bin/ssh-keygen
-rwxr-xr-x root/root    136784 2002-03-13 01:32:58 bin/ssh-keyscan
-rwxr-xr-x root/root     46216 2002-03-13 01:32:58 bin/sftp
lrwxrwxrwx root/root         0 2002-03-13 01:32:59 bin/slogin -> ssh
-rwxr-xr-x root/root    230416 2002-03-13 01:32:58 sbin/sshd
-rw-r--r-- root/root       182 2002-03-13 02:31:06 sbin/generate-ssh-keys
-rw-r--r-- root/root     46525 2002-03-13 01:32:58 man/man1/ssh.1
-rw-r--r-- root/root      3261 2002-03-13 01:32:59 man/man1/scp.1
-rw-r--r-- root/root      4896 2002-03-13 01:32:59 man/man1/ssh-add.1
-rw-r--r-- root/root      6315 2002-03-13 01:32:59 man/man1/ssh-agent.1
-rw-r--r-- root/root     10081 2002-03-13 01:32:59 man/man1/ssh-keygen.1
-rw-r--r-- root/root      3890 2002-03-13 01:32:59 man/man1/ssh-keyscan.1
-rw-r--r-- root/root      7129 2002-03-13 01:32:59 man/man1/sftp.1
lrwxrwxrwx root/root         0 2002-03-13 01:32:59 man/man1/slogin.1 -> ssh.1
-rw-r--r-- root/root     43258 2002-03-13 01:32:59 man/man8/sshd.8
-rw-r--r-- root/root      2087 2002-03-13 01:32:59 man/man8/sftp-server.8
-rwxr-xr-x root/root     23712 2002-03-13 01:32:58 libexec/sftp-server
-rw-r--r-- root/root       568 2002-03-13 01:32:56 share/Ssh.bin
-rw-r--r-- root/root     88039 2002-03-13 01:32:59 etc/moduli
-rw-r--r-- root/root      1144 2002-03-13 01:32:59 etc/ssh_config
-rw-r--r-- root/root      2523 2002-03-13 01:32:59 etc/sshd_config

The only nonstandard file is sbin/generate-ssh-keys, which is a small shell script we have created: it simply creates the three key pairs that are required. Run this program one time after installation to create these keys.

Host Key Generation

The SSH daemon requires its own set of host keys, and these are created as part of the installation process. But when the binaries are copied to another machine, the keys must be generated manually. As with all keys, these are created with the ssh-keygen program, and the three sets of keys are created with:

# ssh-keygen -t rsa1 -f /usr/local/etc/ssh_host_key -N ""
# ssh-keygen -t dsa -f /usr/local/etc/ssh_host_dsa_key -N ""
# ssh-keygen -t rsa -f /usr/local/etc/ssh_host_rsa_key -N ""

References

[O'Reilly SSH book]

This entire Tech Tip is very lame in comparison with the excellent O'Reilly book SSH, The Secure Shell: The Definitive Guide (2e), which covers everything in much more detail. Once you've gotten a handle on the basic configuration of OpenSSH, we recommend picking up this book to get to the finer points. We'll update our document as we pick them up ourselves.


Upgrading for the ZLIB problem

There has been a problem reported in the ZLIB compression library that has prompted version 1.1.4. We've installed this and found it easy to do.

Portability Notes

We've managed to build OpenSSH on several other platforms with more or less the same instructions for Linux, but a few notes might be helpful here.

SCO UNIX 5.0.6 with SCO cc
This took a while to get around some build issues, but we ultimately got it working just fine.
  • zlib-1.1.4 built and installed without incident
  • openssl-0.9.7 required that we edit the Configure file and find the line for "sco-cc". Remove -lresolv from this line - we didn't find this file on our system, and our attempts to build BIND9 on this platform failed due to the lack of a "long long" integer type. Then build, test, and install per instructions.
  • openssh-3.5p1 fails the "configure" process because the default cc output is COFF, not ELF. So the command to set this up is ./configure --with-ipv4-default CC="cc -belf". Then build and install per the usual instructions.
SCO UNIX 5.0.5 with GNU cc
This built and installed with the above instructions excepting configuring OpenSSH itself (SCO doesn't use MD5 shadow passwords).
# cd openssh-3.2.3p1 (with gcc)
# ./configure --with-ipv4-default
We installed the startup file in /etc/rc2.d/S99sshd. This information built as of:
  • zlib-1.1.4
  • openssl-0.9.6d
  • openssh-3.2.3p1
The sshd privsep user directory must be created manually with all the right permissions lest it complain about the modes.
# mkdir -m 700 /var/empty
# chown sshd /var/empty
# chgrp sshd /var/empty
Sun SOLARIS 2.7
This likewise built with the same procedure with the exception of the the OpenSSH configuration itself:
# cd openssh-3.2.2p1
# ./configure --with-ipv4-default --with-pam
Built with:
  • zlib-1.1.4
  • openssl-0.9.6d
  • openssh-3.2.2p1

Privilege Separation

Starting with OpenSSH 3.3.2, the privilege separation feature was introduced. This is a security provision that goes a long way to forestall full system compromise even if a vulnerability is found. The details are covered elsewhere, and our favorite is here.

But before SSHD is installed, both a sshd user and group must be added, and it looks in the password and group files like this:

/etc/group sshd:*:27:
/etc/passwd sshd:*:27:27:sshd privsep:/var/empty:/sbin/nologin

Under Red Hat Linux 7.2, these commands do the trick:

# groupadd sshd
# useradd -M -g sshd -c 'sshd privsep' -d /var/empty -s /sbin/nologin sshd
# passwd -l sshd

The -M parameter suppresses creation of the home directory, as this is done by the "make install" procedure for OpenSSH.

Note further that the UsePrivilegeSeparation yes option must be set in the sshd_config file - this may be the default for new installations, but upgrades don't usually touch current values. Make sure this option is set.

Related Resources


Feedback on this document is very much welcomed by [Email address].