Safely removing old Linux kernels from Ubuntu

By Luke Tymowski, Systems Administrator, Calgary

Ubuntu 12.04, while a LTS (Long Term Support) release, has been pushing new kernels out the door with surprising frequency (one or two every month). If you're running production servers that you cannot reboot every month, it doesn't take long to fill up your /boot.

Ubuntu, unlike RedHat, does not not include a utility as part of its base installation to safely remove these old kernels. So, you can either remove them by hand (not recommended) or create a script. I used a script to remove all but the latest and latest-1 kernels after rebooting the server. I do this in monthly or bimonthly maintenance windows for our infrastructure servers.

But if you've got servers that you cannot reboot unless it's absolutely necessary, you need to preserve the current and the latest kernel, while removing all the other kernels. We have a few OpenStack-based projects for which this is true.

Early in 2012, there was some discussion about this problem on the Ubuntu developer mailing list, but the discussion petered out without a clear resolution in place.

However, an Ubuntu developer did come up with a script, which he submitted to Bikeshed. (Ubuntu's Bikeshed is where packages hang out until someone reaches out and commits to permanently adopt, feed, and care for them.)

There were two problems with the proposed script:

  • it didn't include a path statement, so when running the script unattended, the script would fail because it couldn't find the system commands.
  • it didn't include the necessary parameters to allow the script to run unattended, so it would try to prompt for user input, then fail.

You should add this path statement to the top of the file:

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Add -yq to:

dpkg-query -s "$c" >/dev/null 2>&1 && PURGE="$PURGE $c"

Leaving you with:

dpkg-query -syq "$c" >/dev/null 2>&1 && PURGE="$PURGE $c"

And edit:

apt-get $APT_OPTS remove --purge $PURGE

Leaving you with:

apt-get $APT_OPTS -yq remove --purge $PURGE

As this was a much better script than what I had been using, after a couple of test runs, I dropped my old script and adopted this one because it keeps the current, running, kernel, and the two latest, but removing all the other kernels.

So, now you've got a script that you need to distribute to 20 or 2,000 servers. And you want to run it automatically once a month to clean up /boot. Since we use both Puppet and Chef at Cybera, below are recipes for both.

Puppet

We put these sorts of scripts into our admin module:
/etc/puppet/modules/admin

I placed the script itself here:
/etc/puppet/modules/admin/files/misc/purge-old-kernels

Created the class file:
/etc/puppet/modules/admin/manifests/misc/scripts.pp

Which consists of:
class admin::misc::scripts {

  file { '/etc/cron.d/purge-old-kernels':
    ensure  => present,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => "5 1 1 * * root /usr/local/sbin/purge-old-kernels > /dev/null 2>&1n",
  }

  file { '/usr/local/sbin/purge-old-kernels':
    ensure => present,
    owner  => 'root',
    group  => 'root',
    mode   => '0744',
    source => "puppet:///modules/admin/misc/purge-old-kernels",
  }

}

In our Puppet installation, we have a base node definition where we specify which packages are installed on all servers. We trigger the installation of the purge-old-kernels script like so:

node basic_server {
  # Add a script to each server to purge old kernels
  # Retaining old kernels causes /boot to fill up
  class { 'admin::misc::scripts': }

And if you use Chef, we've got you covered too. As we do with Puppet, we have a base recipe that is run on all servers:
cookbooks/base/recipes

Edit cookbooks/base/recipes/default.rb and add:
include_recipe "base::scripts"

Edit cookbooks/base/recipes/scripts.rb and add:
template "/usr/local/sbin/purge-old-kernels" do
  source "purge-old-kernels.erb"
  owner "root"
  group "root"
  mode "0744"
end

file "/etc/cron.d/purge-old-kernels" do
  owner "root"
  group "root"
  mode 00744
  content "5 1 1 * * root /usr/local/sbin/purge-old-kernels > /dev/null 2>&1n"
  action :create
end

Place the purge-old-kernels script here and just append .erb to the filename:
cookbooks/base/templates/default/purge-old-kernels.erb

And don't forget to update the version number in:
cookbooks/base/metadata.rb