Episode #44 - Patching the GHOST glibc gethostbyname CVE-2015-0235 bug

Loading the player...

About Episode - Duration: 14 minutes, Published: 2015-01-28

In this episode, we are going to be patching CentOS and Debian systems against glibc gethostbyname function bugs. System library security bugs present a special case, where even through you have patched the bug, you are likely still running vulnerable code, unless you reboot.

Download: mp4 or webm

Get notified about future content via the mailing list, follow @jweissig_ on Twitter for episode updates, or use the RSS feed.

Links, Code, and Transcript


In this episode, we are going to be patching CentOS and Debian systems against glibc gethostbyname function bugs. System library security bugs present a special case, where even through you have patched the bug, you are likely still running vulnerable code, unless you reboot.

Overview

On January 27th 2015, an announcement went out about a security issue in glibc gethostbyname set of functions. Many Linux distributions are affected by this issue, and one thing is clear, this is both remotely, and locally exploitable, by application that do DNS resolving against glibc gethostbyname function set. If you are running RHEL, or CentOS up to and including version 7, then you need to patch, same goes for Debian 7, and Ubuntu 12.04. In this announcement, there is even mention of a proof of concept exploit, that spawns a remote shell, via a specially crafted email. There is a fantastic technical analysis of the issue, linked to in the episode notes below, if you are wanting to get into the guts of the bug. In fact, I have also included all my research links in the episode notes below, so hopefully you will find them useful.

Since there is so much good information already out there about the bug, I wanted to focus on the patch workflow, and how to verify you are safe. It is important to patch your systems, but if you do not reboot after the patch, then I wanted to explain why you need to restart applications which touch glibc gethostbyname, because they could be linked to an older vulnerable version of a gethostbyname function based in memory. I plan to show you how to check, and verify, that each of your systems is up to date. The fail safe method, would be to install the patches, and reboot, but for many this is not feasible, so I wanted to show you a potential gotcha. My hope is that, this will be generic enough to serve as a pattern, which can be applied to all types of system library bugs.

For the demo today, I have fired up two virtual machines, a CentOS 7 box, and a Debian 7.4 Wheezy box. We are going to step through the process of patching both of these machines, then I will talk about why this issue is complex, since we are dealing with a system library.

Before we start the demos, I should mention that I am using Vagrant to launch these virtual machines, and have included the Vagrantfile in the episode transcript below. If you do not know about Vagrant, it is a type of wrapper around virtualization software, that allows you to automate much of the manual work associated with launching test environments. Check out episode #42, where I give a Crash Course on Vagrant, it is pretty cool.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  # centos 
  config.vm.define :centos do |node|
      node.vm.box = "chef/centos-7.0"
      node.vm.hostname = "centos-7"
      node.vm.provider "virtualbox" do |vb|
        vb.memory = "256"
      end
  end

  # debian
  config.vm.define :debian do |node|
      node.vm.box = "chef/debian-7.4"
      node.vm.hostname = "debian-74"
      node.vm.provider "virtualbox" do |vb|
        vb.memory = "256"
      end
  end

end

So, in the first terminal tab here, I have a CentOS 7 machine loaded, and in the second tab, a Debian 7.4 Wheezy box. We are going to work through the process of checking our systems, installing the patch, then verifying the patch works as expected. We will also investigate why a reboot is likely important, because many application launched before we installed the patch, will still be linked against the buggy glibc stored in memory. I found this great blog post by Mattias about the issue, and these couple paragraphs here, inspired me to create this episode. There is also a pretty good thread on Hacker News about this too, where I suggested using lsof via Mattias blog post, Josh Wright had a nice follow up about using grep to filter the information down. I just wanted to compile all of my research, and tests into one place, so that I could share them. Links are in the episode notes section below.

Patching CentOS patching CVE-2015-0235

Lets start the demo using our CentOS 7 box. Let verify we are running a vulnerable version of the glibc package, by running rpm query info glibc. We have glibc version 2.17, here is the redhat release, this will change when we upgrade, and this package was built December 18, 2014. This gives us our starting point, something to compare against. You can checkout the linked to security announcement from both Redhat and Debian to see if you are vulnerable. There was an example c program, from the technical analysis posting, which you can copy to your local machine, compile, then run, to see if your machine is vulnerable. I have already done these steps on both our virtual machines today, and this is what this ghost-64bit binary is. Running it returns a vulnerable notice.

[vagrant@centos-7 ~]$ cat /etc/redhat-release 
CentOS Linux release 7.0.1406 (Core) 
[vagrant@centos-7 ~]$ rpm --query --info glibc
Name        : glibc
Version     : 2.17
Release     : 55.el7_0.3
Architecture: x86_64
Install Date: Wed 28 Jan 2015 04:51:21 AM UTC
Group       : System Environment/Libraries
Size        : 13953497
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Fri 19 Dec 2014 12:39:36 PM UTC, Key ID 24c6a8a7f4a80eb5
Source RPM  : glibc-2.17-55.el7_0.3.src.rpm
Build Date  : Thu 18 Dec 2014 11:02:33 PM UTC
[vagrant@centos-7 ~]$ ls -l
-rwxr-xr-x. 1 vagrant vagrant 6088 Jan 28 04:45 ghost-64bit
[vagrant@centos-7 ~]$ ./ghost-64bit 
vulnerable
[vagrant@centos-7 ~]$ sudo yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors

Okay, so lets update our system, by running sudo yum clean all. I am not sure if this step is required, but generally I like to clean the package cache. Then, lets run sudo yum update to get the latest packages. Great, looks like there is an updated to glibc. Yes, I want to install them. If it has been a while since you installed patches, you might want to review if there is anything else included in the listing. Great, our updates are complete.

[vagrant@centos-7 ~]$ sudo yum update

...
Resolving Dependencies
--> Running transaction check
---> Package glibc.x86_64 0:2.17-55.el7_0.3 will be updated
---> Package glibc.x86_64 0:2.17-55.el7_0.5 will be an update
---> Package glibc-common.x86_64 0:2.17-55.el7_0.3 will be updated
---> Package glibc-common.x86_64 0:2.17-55.el7_0.5 will be an update
--> Finished Dependency Resolution

...

Updated:
  glibc.x86_64 0:2.17-55.el7_0.5
  glibc-common.x86_64 0:2.17-55.el7_0.5

Complete!

Let verify we have a fixed version of the glibc package, by running rpm query info glibc again. We have glibc version based on 2.17, here is the updated Redhat release, you might have noticed that before this ended in a dot 3, now it is dot 5, and the updated package build date is January 27, 2015, and this is the one we want based off the vendor security announcement.

[vagrant@centos-7 ~]$ rpm --query --info glibc
Name        : glibc
Version     : 2.17
Release     : 55.el7_0.5
Architecture: x86_64
Install Date: Wed 28 Jan 2015 05:05:02 AM UTC
Group       : System Environment/Libraries
Size        : 13953497
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Wed 28 Jan 2015 12:01:05 AM UTC, Key ID 24c6a8a7f4a80eb5
Source RPM  : glibc-2.17-55.el7_0.5.src.rpm
Build Date  : Tue 27 Jan 2015 08:13:46 PM UTC

But, lets check out the rpm changelog for this updated glibc package, just to be sure it includes the changes we are looking for. We can run rpm query changelog glibc, and pipe the output to head, since our changes are most likely at the top of the file, you could use something like less too. As you can see here, this changelog mentions that it includes the fix we are looking for, here is the CVE number, and the Redhat Bugzilla number.

So, at this point, it looks like we are in good shape, but lets verify our vulnerability checker from the security announcement agrees that our system is safe. Great, looks like our system is patched, since the package has been updated, and this returns not vulnerable.

[vagrant@centos-7 ~]$ rpm --query --changelog glibc |head
* Mon Jan 19 2015 Carlos O'Donell <codonell@redhat.com> - 2.17-55.5
- Rebuild and run regression testing.

* Mon Jan 19 2015 Siddhesh Poyarekar <siddhesh@redhat.com> - 2.17-55.4
- Fix parsing of numeric hosts in gethostbyname_r (CVE-2015-0235, #1183535).

* Fri Dec 05 2014 Carlos O'Donell <carlos@redhat.com> - 2.17-55.3
- Fix wordexp() to honour WRDE_NOCMD (CVE-2014-7817, #1170118)

* Thu Dec 04 2014 Siddhesh Poyarekar <siddhesh@redhat.com> - 2.17-55.2

But is this really fixed? Well, this is that potential gotcha I wanted to show you. Sorry that it took so long, but I wanted to set the stage so that we could play around. The issue is that, all of the launched and running processes before our glibc update, are still using the older buggy version of glibc stored in ram. What I mean, is that when you launch something like a web server, that is linked against glibc, it will cache that library in memory. So, even though we updated glibc on disk, many of the existing processes on our system are still using the buggy glibc cached in memory. You might be wondering why, the ghost test script actually return not vulnerable, and that is because when it runs, it starts and entirely new process, post glibc update, and that checks against the latest version of glibc from disk.

[vagrant@centos-7 ~]$ ls -l
-rwxr-xr-x. 1 vagrant vagrant 6088 Jan 28 04:45 ghost-64bit
[vagrant@centos-7 ~]$ ./ghost-64bit 
not vulnerable

An easy fix, is to just reboot the box, but there are many situations where you need to schedule maintenance windows. How do you find these processes linked against the buggy version of glibc, so that we can manually restart them? Well, that is where Mattias lsof command suggestion come in, where we grep for libc, and print out all of the process names that likely link against it. This should give us a good idea of what we are dealing with. Okay, as you might expect, the list is actually pretty long. Since this glibc bug is remotely exploitable, lets verify some of our network connected services are actually updated, like we think they are. Lets pick on the postfix mail server here, this master process is part of the postfix mail package. Lets find the process id, using pid of, then lets run lsof, let it know that we want to focus on a specific process id, then pass in our 1074 process number. Whoa, there is lots of output there. Lets filter that down, by piping this to grep, and matching about DEL. DEL just so happens to be a lsof type for processes with a deleted memory map. So, what you are looking at here, is a network connected postfix process, linked against old lib files, and you can see libc here. So, I guess what I am getting at, is that even though we verified we are running the latest glibc package, and our vulnerability tester script says we are okay, this does not in fact mean we are okay. I am not sure if postfix is exploitable or not, but I think there is a false sense of security out there, that all you need to do is patch, and you are safe. This is not the case, because as you can see, we have patched our system, but a network connected process could be exploitable still. You need to either restart all of these process linked against the buggy glibc, I would focus on network connected ones first, or a much easier solution, is to probably just reboot.

[vagrant@centos-7 ~]$ sudo lsof | grep libc | awk '{print $1}' | sort | uniq
...
master
...
sshd
...
[vagrant@centos-7 ~]$ pidof master
1074
[vagrant@centos-7 ~]$ sudo lsof -p 1074
COMMAND  PID USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
...
master  1074 root  DEL       REG              253,1          134787665 /usr/lib64/libc-2.17.so
...
[vagrant@centos-7 ~]$ sudo lsof -p 1074 |grep DEL
master  1074 root  DEL       REG              253,1          134801283 /usr/lib64/libnss_files-2.17.so
master  1074 root  DEL       REG              253,1          134364091 /usr/lib64/libfreebl3.so
master  1074 root  DEL       REG              253,1          134296680 /usr/lib64/libgcc_s-4.8.2-20140120.so.1;54c86ac3
master  1074 root  DEL       REG              253,1          134801295 /usr/lib64/librt-2.17.so
master  1074 root  DEL       REG              253,1          134787669 /usr/lib64/libcrypt-2.17.so
master  1074 root  DEL       REG              253,1          134801315 /usr/lib64/libstdc++.so.6.0.19
master  1074 root  DEL       REG              253,1          134787671 /usr/lib64/libdl-2.17.so
master  1074 root  DEL       REG              253,1          134801291 /usr/lib64/libpthread-2.17.so
master  1074 root  DEL       REG              253,1          134373189 /usr/lib64/libnspr4.so
master  1074 root  DEL       REG              253,1          134373190 /usr/lib64/libplc4.so
master  1074 root  DEL       REG              253,1          134373191 /usr/lib64/libplds4.so
master  1074 root  DEL       REG              253,1          134373208 /usr/lib64/libnssutil3.so
master  1074 root  DEL       REG              253,1          134602724 /usr/lib64/libnss3.so
master  1074 root  DEL       REG              253,1          134602733 /usr/lib64/libsmime3.so
master  1074 root  DEL       REG              253,1          134602734 /usr/lib64/libssl3.so
master  1074 root  DEL       REG              253,1          134787665 /usr/lib64/libc-2.17.so
master  1074 root  DEL       REG              253,1          134801293 /usr/lib64/libresolv-2.17.so
master  1074 root  DEL       REG              253,1          134787675 /usr/lib64/libnsl-2.17.so
master  1074 root  DEL       REG              253,1          134373204 /usr/lib64/libdb-5.3.so
master  1074 root  DEL       REG              253,1          134637707 /usr/lib64/libcrypto.so.1.0.1e
master  1074 root  DEL       REG              253,1          134637708 /usr/lib64/libssl.so.1.0.1e
master  1074 root  DEL       REG              253,1          134787673 /usr/lib64/libm-2.17.so
master  1074 root  DEL       REG              253,1          201651245 /usr/lib64/mysql/libmysqlclient.so.18.0.0;54c86ac3
master  1074 root  DEL       REG              253,1          134787658 /usr/lib64/ld-2.17.so

This is not a new phenomenon, and there are actually tools which make this a little easier to deal with, rather than lugging through lsof output. On RHEL and its derivatives, you can use the needs-restarting utility, but it is not installed by default. Lets install the yum utils package, which includes needs-restarting, so that we can play around with it.

Now that we have it installed, lets run sudo needs-restarting, which will hopefully tell us all of the running processes linked again old libs, that need restarting. As you can see, we have a much prettier output, giving us the process ids, and command paths. Personally, I find this a much more intuitive view into the problem, and this allows you to go through the cycle to restarting apps as needed.

[vagrant@centos-7 ~]$ sudo yum install yum-utils.noarch

Resolving Dependencies
--> Running transaction check
---> Package yum-utils.noarch 0:1.1.31-25.el7_0 will be installed
--> Processing Dependency: python-kitchen for package: yum-utils-1.1.31-25.el7_0.noarch
--> Running transaction check
---> Package python-kitchen.noarch 0:1.1.1-5.el7 will be installed
--> Processing Dependency: python-chardet for package: python-kitchen-1.1.1-5.el7.noarch
--> Running transaction check
---> Package python-chardet.noarch 0:2.0.1-7.el7 will be installed
--> Finished Dependency Resolution

...

Installed:
  yum-utils.noarch 0:1.1.31-25.el7_0                                                                     

Dependency Installed:
  python-chardet.noarch 0:2.0.1-7.el7
  python-kitchen.noarch 0:1.1.1-5.el7

Complete!
[vagrant@centos-7 ~]$ sudo needs-restarting
...
854 : /usr/sbin/sshd -D 
...
1074 : /usr/libexec/postfix/master -w 
...
[vagrant@centos-7 ~]$ ls -l
-rwxr-xr-x. 1 vagrant vagrant 6088 Jan 28 04:45 ghost-64bit
[vagrant@centos-7 ~]$ ./ghost-64bit 
not vulnerable

However, lets just reboot the system, so that we can see what a clean slate looks like, and through the magic of video editing we are back. Lets just verify we are in an okay state, lets run our vulnerability checker script, and lets view the package information again. Okay, so it looks like our system is patched. Lets do the final test, and run needs-restarting again, and there is nothing returned. Lets also check out postfix process again, by running sudo lsof, specify that we want to review a process id, and get the pidof master, and pipe this to less. If we search for instances of libc, you will see that there is a mem type, where we used to have DEL type. This just means, that we have a memory mapped library, where before the memory mapped library was deleted, due to our software update.

[vagrant@centos-7 ~]$ sudo reboot
[vagrant@centos-7 ~]$ uptime
 05:15:53 up 0 min,  1 user,  load average: 0.73, 0.16, 0.05
[vagrant@centos-7 ~]$ ls -l
-rwxr-xr-x. 1 vagrant vagrant 6088 Jan 28 04:45 ghost-64bit
[vagrant@centos-7 ~]$ ./ghost-64bit 
not vulnerable
[vagrant@centos-7 ~]$ rpm --query --info glibc
Name        : glibc
Version     : 2.17
Release     : 55.el7_0.5
Architecture: x86_64
Install Date: Wed 28 Jan 2015 05:05:02 AM UTC
Group       : System Environment/Libraries
Size        : 13953497
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Wed 28 Jan 2015 12:01:05 AM UTC, Key ID 24c6a8a7f4a80eb5
Source RPM  : glibc-2.17-55.el7_0.5.src.rpm
Build Date  : Tue 27 Jan 2015 08:13:46 PM UTC
[vagrant@centos-7 ~]$ sudo needs-restarting
[vagrant@centos-7 ~]$ sudo lsof -p `pidof master`|less
COMMAND  PID USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
...
master  1055 root  mem       REG              253,1  2107600 134787645 /usr/lib64/libc-2.17.so
...

So, I guess the lesson of the story, is that when dealing with library security bugs, do not trust you are in a safe state until you reboot, as there is likely running processed using the vulnerable code. This has themes into the OpenSSL Heartbleed bug too, so I just wanted to put something detailed together, which will hopefully explain the process in depth.

Patching Debian patching CVE-2015-0235

I thought it might make sense to cover Debian too, since this applies to Ubuntu 12.04 as well, and it just make for a thorough look at the tools they use to find processes using older version of libs. So, lets switch tabs, to our Debian 7.4 Wheezy box.

vagrant@debian-74:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.4 (wheezy)
Release:    7.4
Codename:   wheezy

Lets check the version of the libc6 package, you will notice that Debian and Ubuntu use the libc6 package name, rather than glibc, just something to keep in mind. The version installed is vulnerable, check out the associated vendor security announcements in linked to in the episode nodes below.

vagrant@debian-74:~$ dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 9522
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.13-38+deb7u1
Replaces: libc6-amd64
Provides: glibc-2.13-1

Again, we can run the ghost vulnerability checker tool, and it says we are vulnerable. So, lets update our system, by running sudo apt-get update, this will download the lastest package metadata.

vagrant@debian-74:~$ ls -l
-rwxr-xr-x 1 vagrant vagrant 6088 Jan 28 05:56 ghost-64bit
vagrant@debian-74:~$ ./ghost-64bit 
vulnerable

Then we can run sudo apt-get upgrade, this will upgrade the packages on our system to the latest levels. You will notice that on my test system here, I have not run updates in a while, so there are many packages to install. After a few minutes, those have all downloaded, and installed correctly. Lets verify that our libc6 package is at the correct revision level now. This version, matches what was in the vendor patch announcement message, so I am pretty confident at this point, but lets run our ghost vulnerability checker tool again, and this time, it says we are not vulnerable.

vagrant@debian-74:~$ sudo apt-get update
Get:1 http://mirrors.kernel.org wheezy Release.gpg [1,655 B]
...
Get:17 http://mirrors.kernel.org wheezy-updates/main Translation-en [2,429 B]
Fetched 16.6 MB in 8s (2,012 kB/s)                                                                      
Reading package lists... Done
vagrant@debian-74:~$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  acpi-support-base apt apt-utils at base-files bash bind9-host binutils bsd-mailx ca-certificates cpio
  curl debian-archive-keyring dnsutils dpkg exim4 exim4-base exim4-config exim4-daemon-light file gnupg
  gpgv host krb5-locales libapt-inst1.5 libapt-pkg4.12 libbind9-80 libc-bin libc-dev-bin libc6
  libc6-dev libcurl3 libdns88 libevent-2.0-5 libgcrypt11 libgnutls26 libgpgme11 libgssapi-krb5-2
  libisc84 libisccc80 libisccfg82 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 liblwres80
  libmagic1 libssl1.0.0 libtasn1-3 libxml2 linux-headers-3.2.0-4-amd64 linux-headers-3.2.0-4-common
  linux-image-3.2.0-4-amd64 linux-libc-dev locales mime-support multiarch-support mutt openssh-client
  openssh-server openssl perl perl-base perl-modules procmail python-reportbug reportbug rsyslog tzdata
  wget
70 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 82.9 MB of archives.
After this operation, 550 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
...
vagrant@debian-74:~$ dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 9529
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.13-38+deb7u7
Replaces: libc6-amd64
Provides: glibc-2.13-1
vagrant@debian-74:~$ ls -l 
-rwxr-xr-x 1 vagrant vagrant 6088 Jan 28 05:56 ghost-64bit
vagrant@debian-74:~$ ./ghost-64bit 
not vulnerable

I should mention that I tried to check the libc6 changelog, by running sudo apt-get changelog libc6, but I received a 404 not found error message, saying the changelog download failed. I am not sure if this is because the package was just released or what, but it looks like we cannot review the changelog like we did on our CentOS node.

vagrant@debian-74:~$ sudo apt-get changelog libc6
Err Changelog for libc6 (http://packages.debian.org/changelogs/pool/updates/main/e/eglibc/eglibc_2.13-38+deb7u7/changelog)
  404  Not Found [IP: 5.153.231.3 80]
Err Changelog for libc6 (http://security.debian.org/pool/updates/main/e/eglibc/eglibc_2.13-38+deb7u7.changelog)
  404  Not Found [IP: 149.20.20.6 80]
E: changelog download failed

At this point, you might suspect that your system is patched against this nasty bug, but you likely know better now. So, lets run Mattias lsof command again, grepping for libc, and also for anything matching DEL, since we now know that means libs with a missing memory map.

vagrant@debian-74:~$ sudo lsof | grep libc | grep DEL | awk '{print $1}' | sort | uniq
automount
bash
control
cpuhotplu
cron
dhclient
getty
memballoo
rpcbind
rpc.idmap
rpc.statd
sshd
timesync
udevd
VBoxServi
vminfo
vmstats

This list looks a little shorter than our CentOS box, but there are still some network connected processes, which are running vulnerable glibc code. Lets pick on rpcbind in this case, I should mention that I do not know if any of these are actually exploitable, just that this is something you want to watch out for. So, lets run sudo lsof, specify a process id, and tell it we want to use the pidof rpcbind.

vagrant@debian-74:~$ sudo lsof -p `pidof rpcbind`
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
rpcbind 1618 root  cwd    DIR              254,0     4096      2 /
rpcbind 1618 root  rtd    DIR              254,0     4096      2 /
rpcbind 1618 root  txt    REG              254,0    48080 130974 /sbin/rpcbind
rpcbind 1618 root  DEL    REG              254,0          915732 /lib/x86_64-linux-gnu/libnss_files-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915730 /lib/x86_64-linux-gnu/libdl-2.13.so
rpcbind 1618 root  mem    REG              254,0    39192 919513 /lib/x86_64-linux-gnu/libgssglue.so.1.0.0
rpcbind 1618 root  DEL    REG              254,0          915726 /lib/x86_64-linux-gnu/libnsl-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915715 /lib/x86_64-linux-gnu/libc-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915734 /lib/x86_64-linux-gnu/libpthread-2.13.so
rpcbind 1618 root  mem    REG              254,0   158280 919523 /lib/x86_64-linux-gnu/libtirpc.so.1.0.10
rpcbind 1618 root  mem    REG              254,0    40656 919525 /lib/x86_64-linux-gnu/libwrap.so.0.7.6
rpcbind 1618 root  DEL    REG              254,0          915738 /lib/x86_64-linux-gnu/ld-2.13.so
rpcbind 1618 root    0u   CHR                1,3      0t0   1205 /dev/null
rpcbind 1618 root    1u   CHR                1,3      0t0   1205 /dev/null
rpcbind 1618 root    2u   CHR                1,3      0t0   1205 /dev/null
rpcbind 1618 root    3r   REG               0,14        0   5035 /run/rpcbind.lock
rpcbind 1618 root    4u  sock                0,7      0t0   5059 can't identify protocol
rpcbind 1618 root    5u  unix 0xffff88000bd4f400      0t0   5036 /var/run/rpcbind.sock
rpcbind 1618 root    6u  IPv4               5038      0t0    UDP *:sunrpc 
rpcbind 1618 root    7u  IPv4               5041      0t0    UDP *:945 
rpcbind 1618 root    8u  IPv4               5042      0t0    TCP *:sunrpc (LISTEN)
rpcbind 1618 root    9u  IPv6               5045      0t0    UDP *:sunrpc 
rpcbind 1618 root   10u  IPv6               5048      0t0    UDP *:945 
rpcbind 1618 root   11u  IPv6               5049      0t0    TCP *:sunrpc (LISTEN)

As you can see, we are indeed linked against older version of the libc library, denoted by this DEL, compared to a current library, denoted by the mem type. You could also grep by DEL for a consolidated view if the output is huge.

vagrant@debian-74:~$ sudo lsof -p `pidof rpcbind`|grep DEL
rpcbind 1618 root  DEL    REG              254,0          915732 /lib/x86_64-linux-gnu/libnss_files-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915730 /lib/x86_64-linux-gnu/libdl-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915726 /lib/x86_64-linux-gnu/libnsl-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915715 /lib/x86_64-linux-gnu/libc-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915734 /lib/x86_64-linux-gnu/libpthread-2.13.so
rpcbind 1618 root  DEL    REG              254,0          915738 /lib/x86_64-linux-gnu/ld-2.13.so

Okay, so on our CentOS node, we used the needs-restarting tool, to tell us information about processed that needed to be restarted. Well, Debian has a similar tool, called checkrestart. To, use it, you also need to install an additional package, so lets to that now, by running sudo apt-get install debian-goodies.

vagrant@debian-74:~$ sudo apt-get install debian-goodies
Reading package lists... Done
...
Unpacking debian-goodies (from .../debian-goodies_0.61_all.deb) ...
Processing triggers for man-db ...
Setting up dctrl-tools (2.22.2) ...
Setting up debian-goodies (0.61) ...

Once that is all done, sudo checkrestart -v, and this will tell us all of the currently running processes using deleted files, and as you can see there is tons of output.

vagrant@debian-74:~$ sudo checkrestart -v
Found 17 processes using old versions of upgraded files
(10 distinct programs)
Process /sbin/dhclient (PID: 2713) 
List of deleted files in use:
    /lib/x86_64-linux-gnu/libnss_files-2.13.so
    /lib/x86_64-linux-gnu/libc-2.13.so
    /lib/x86_64-linux-gnu/ld-2.13.so
Process /usr/sbin/sshd (PID: 2805) 
List of deleted files in use:
    /usr/sbin/sshd (deleted)
    /lib/x86_64-linux-gnu/libnss_files-2.13.so
    /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    /lib/x86_64-linux-gnu/libpthread-2.13.so
    /lib/x86_64-linux-gnu/libresolv-2.13.so
    /lib/x86_64-linux-gnu/libkeyutils.so.1.4
    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
    /lib/x86_64-linux-gnu/libdl-2.13.so
    /lib/x86_64-linux-gnu/libnsl-2.13.so
    /lib/x86_64-linux-gnu/libc-2.13.so
    /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
    /lib/x86_64-linux-gnu/libcrypt-2.13.so
    /lib/x86_64-linux-gnu/libutil-2.13.so
    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    /lib/x86_64-linux-gnu/ld-2.13.so
...

So, lets just run the same command, but this time, piping the output into less. As you can see, there are 17 processes using old versions of upgraded files, things like out dhcp client, rpc statd, and sshd. Again, if you cannot reboot, I would focus on processes that are network connected, but to be honest, a reboot is likely the best course of action. So, lets reboot this machine.

vagrant@debian-74:~$ sudo checkrestart -v|less

And through the magic of video editing we are back up, lets just verify our package is still at the correct version, and that our vulnerability check tool, say we are in good shape. Now, that we have rebooted, lets run sudo checkrestart again, and see if there are any issues we need to fix. Nothing, so as you can see, a reboot is highly recommended after security issues with glibc.

vagrant@debian-74:~$ sudo reboot
vagrant@debian-74:~$ uptime
 06:10:46 up 0 min,  1 user,  load average: 0.00, 0.00, 0.00
vagrant@debian-74:~$ dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 9529
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.13-38+deb7u7
Replaces: libc6-amd64
Provides: glibc-2.13-1
vagrant@debian-74:~$ ls -l
-rwxr-xr-x 1 vagrant vagrant 6088 Jan 28 05:56 ghost-64bit
vagrant@debian-74:~$ ./ghost-64bit 
not vulnerable
vagrant@debian-74:~$ sudo checkrestart -v
Found 0 processes using old versions of upgraded files

Closing Remarks

Hopefully you found this episode interesting, it was sort of spur of the moment, as I have doing tons of research on my own to see the impact on systems I look after. Needs-restarting, and checkrestart, are very useful tools for sorting issues like this out, so hopefully you found this episode informative. If you are looking at mass patching solutions, I highly suggest watching my episodes on configuration management tools, things like Ansible, and Puppet.