#18 - Managing Iptables with Puppet

About Episode - Duration: 13 minutes, Published: 2013-11-12

In this episode, I wanted to show you the Puppet Labs Firewall module. We are going to use what we learned in episode #8, about Vagrant and Puppet, and apply that to managing iptables firewall rules with Puppet.

Download: mp4 or webm

Get notified about future content via the mailing list, follow @jweissig_ on Twitter for episode updates, or use the RSS feed.

Links, Code, and Transcript


In this episode, I wanted to show you the Puppet Labs Firewall module. We are going to use what we learned in episode #8, about Vagrant and Puppet, and apply that to managing iptables firewall rules with Puppet. The Puppet Labs Firewall module is well rounded, seems mature, and is widely deployed, with close to 100 thousand downloads.

If you have ever had to manually manage iptables rules across many Linux hosts, you will know how painful and unwieldy it can get. Lets take a look at an example without puppet for a minute.

Lets assume you have several large clusters, web, app, compute, some database nodes, and many one off machines. These one off nodes could be for internal monitoring, source control, analytics, you name it. If you wanted to run iptables on each of these nodes, how would you do it? Well, you would likely have a base or standard iptables script, and then have many variations based off your systems requirements, it would be a management nightmare, and there is no clear way to verify rules are running as you expect.

On the flip side, with Puppet and the Puppet Labs Firewall module we can get some relief, in that it is really easy to manage many different iptables firewall configurations, and handling exceptions is a breeze. I am not going to lie, it took me a while to wrap my head around how Puppet would manage the firewalls, but once I did some testing, it was surprisingly easy and intuitive.

Back in episode #8, which I have linked to in the episode nodes below, we looked at configuring a virtual machine running CentOS 6.4 via Vagrant. The reason I am mentioning this, is because we are going to use the exact same setup, in this episode. If you are interesting in playing around with Puppet at all, I highly recommend watching episode #8.

Okay, so lets get on with it. You can use this command to install the module, but since I am running Vagrant and a standalone puppet agent, I am just going to copy this link to download the tar ball on the console with wget. I am just going to extract this module and place it into my puppet modules folder.

Lets fire up our editor and take a look. Alright, so you an see my puppet directory hierarchy here. Under the manifests directory, can you see my node definition, and then my modules under the modules directory. You will notice that the Puppet Labs Firewall module is listed here too.

# .../puppet/manifests/site.pp

  # iptables purge

  resources { "firewall":
    purge   => true
  }

  Firewall {
    before  => Class['fw::post'],
    require => Class['fw::pre'],
  }

  class { ['fw::pre', 'fw::post']: }

Lets open up the node definition and have a look. Here we have three blocks that talk about our firewall. The first block, purges any existing iptables rules. This essentially wipes the slate clean so that we can add rules with puppet. The next block uses the Puppet Labs Firewall module which referrers to a custom module called fw, and that fw module has pre and post subclasses, and the logic dependency is that we run the pre class before the post class, then down here, we call the pre and post subclasses.

Lets have a look at the fw module, specifically the pre, and post files. Let me just walk through these files block by block and tell you what is happening. This comment just tells you where this file sits in the directory hierarchy, I mainly added this for you, because I have added these code snippets to the episode transcript below, so that you can copy them if you wish.

# .../puppet/modules/fw/manifests/pre.pp

class fw::pre {
  
  Firewall {
    require => undef,
  }

  # basic in/out

  firewall { "000 accept all icmp":
    chain    => 'INPUT',
    proto    => 'icmp',
    action   => 'accept',
  }

  firewall { '001 accept all to lo interface':
    chain    => 'INPUT',
    proto    => 'all',
    iniface  => 'lo',
    action   => 'accept',
  }

  firewall { '003 accept related established rules':
    chain    => 'INPUT',
    proto    => 'all',
    state    => ['RELATED', 'ESTABLISHED'],
    action   => 'accept',
  }

  firewall { '004 accept related established rules':
    chain    => 'OUTPUT',
    proto    => 'all',
    state    => ['RELATED', 'ESTABLISHED'],
    action   => 'accept',
  }

  # extra

  firewall { '200 allow outgoing icmp type 8 (ping)':
    chain    => 'OUTPUT',
    proto    => 'icmp',
    icmp     => 'echo-request',
    action   => 'accept',
  }  
  
  firewall { '200 allow outgoing dns lookups':
    chain    => 'OUTPUT',
    state    => ['NEW'],
    dport    => '53',
    proto    => 'udp',
    action   => 'accept',
  }  

  firewall { '200 allow outgoing http':
    chain    => 'OUTPUT',
    state    => ['NEW'],
    dport    => '80',
    proto    => 'tcp',
    action   => 'accept',
  }  

}

Next we define the fw pre subclass. The reason we have pre and post subclasses, is that it allows us to specify allow rules in the pre class and then in the post class we can add our deny rules.

The first block here allows incoming icmp packets, things like ping, to be accepted by this host. You will notice that this line acts like a comment, and we can control the order in which rules are executed by this three digit number, in this case it is 000. The next rule accepts all traffic from localhost, along with its comment and three digit sequence number. I should mention that these sequence number can be the same, it just means that the rules might not be in the exact same order each time, but if you have a lot of general access rules, it might make sense to pick a number and just assign them the same code.

The next block here, says accept all related or established connections. Basically, this just means that the kernel will keep track of open connections and allow them.

This next one is a little bit interesting, generally by default iptables is configured to only block incoming traffic going to the INPUT chain, but this rule says, accept all related or established connections in the OUTPUT chain, that is because I have configured this firewall to only allow accepting incoming and outgoing connections. This will make more sense when we review the deny rules in the fw post subclass.

So, that is basically it for the standard rules. Then down here, since I have a firewall rule blocking outgoing connections too, we need to add some rules to allow outgoing traffic. This rules says, allow outgoing icmp ping, then this one allows outgoing dns lookups, and finally, we allow outgoing http connections, this is useful for updates, or fetching packages.

So now that we know about the pre subclass, lets have a look at the post subclass. Again, a comment about where this sits in the hierarchy. The first block here, says to log dropped packets for the INPUT chain, actually, these three rules are very similar, in that they all have the same sequence number, and they all preform the same action, that being they log dropped packets for the INPUT, FORWARD, and OUTPUT chains.

# .../puppet/modules/fw/manifests/post.pp

class fw::post {
  
  firewall { '900 log dropped input chain':
    chain      => 'INPUT',
    jump       => 'LOG',
    log_level  => '6',
    log_prefix => '[IPTABLES INPUT] dropped ',
    proto      => 'all',
    before     => undef,
  }

  firewall { '900 log dropped forward chain':
    chain      => 'FORWARD',
    jump       => 'LOG',
    log_level  => '6',
    log_prefix => '[IPTABLES FORWARD] dropped ',
    proto      => 'all',
    before     => undef,
  }

  firewall { '900 log dropped output chain':
    chain      => 'OUTPUT',
    jump       => 'LOG',
    log_level  => '6',
    log_prefix => '[IPTABLES OUTPUT] dropped ',
    proto      => 'all',
    before     => undef,
  }

  firewall { "910 deny all other input requests":
    chain      => 'INPUT',
    action     => 'drop',
    proto      => 'all',
    before     => undef,
  }

  firewall { "910 deny all other forward requests":
    chain      => 'FORWARD',
    action     => 'drop',
    proto      => 'all',
    before     => undef,
  }

  firewall { "910 deny all other output requests":
    chain      => 'OUTPUT',
    action     => 'drop',
    proto      => 'all',
    before     => undef,
  }

}

Next, we have three matching rules that drop all traffic not matched in an earlier rule. You will notice that they all have the same sequence number too, and these are for the for the INPUT, FORWARD, and OUTPUT chains too.

Okay, so now that we know a little about how these firewall rules work, lets have a look at the node definition again. So, we purge the existing firewall rules, then we use the Puppet Labs Firewall module which referrers to our custom modules called fw, and that fw module has pre and post subclasses.

At this point, you might be wondering where application specific rules go. Say for example, where would we add a rules to allow ssh on each host, or a subset of hosts? Well, lets have a look at the openssh module here. This is a very basic module using the package, file, service pattern. You will notice a firewall rule, here, but let me come back to that is a second. First we install openssh, next we define a config file, and then we make sure sshd is started. So, lets have a second look at this firewall rule, it is basically saying, allow incoming ssh connections, and this rule is within the ssh module. What is so cool about this, is that as you include modules in the host definition, the firewall rules go along too.

# .../puppet/modules/openssh/manifests/init.pp
# the package, file, service pattern


class openssh {

  # firewall logic

  firewall { '100 allow openssh':
    chain   => 'INPUT',
    state   => ['NEW'],
    dport   => '22',
    proto   => 'tcp',
    action  => 'accept',
  }  
  
  package { 'openssh-server':
    ensure  => present,
    before  => File['/etc/ssh/sshd_config'],
  }
  
  file { '/etc/ssh/sshd_config':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0600',
    source  => "puppet:///modules/openssh/sshd_config",
  }
  
  service { 'sshd':
    ensure     => running,
    enable     => true,
    subscribe  => File['/etc/ssh/sshd_config'],
  }

}

Lets say for example, we have three servers defined in our node definition, server one, two, and three. As we include modules to each node, they will have the services install, configuration files updated, and the firewalls tweaked to allow access! So, openssh and httpd modules are included for server two, and along with the modules we get our iptables firewall rules, and server one just has openssh. Then up here we define a standard set of rules which get deployed to each node. Before we look at a live demo, let me just show you the httpd modules for a minute. You can see the same package, service, file pattern, and then we have our firewall rule here too. These are very simplified examples, but they help to illustrate the power of managing iptables firewall rules with puppet.

# .../puppet/modules/httpd/manifests/init.pp
# the package, file, service pattern


class httpd {

  # firewall logic

  firewall { '100 allow httpd':
    chain   => 'INPUT',
    state   => ['NEW'],
    dport   => '80',
    proto   => 'tcp',
    action  => 'accept',
  }  
  
  package { 'httpd':
    ensure  => present,
    before  => File['/etc/httpd/conf/httpd.conf'],
  }
  
  file { '/etc/httpd/conf/httpd.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    source  => "puppet:///modules/httpd/httpd.conf",
  }
  
  service { 'httpd':
    ensure     => running,
    enable     => true,
    subscribe  => File['/etc/httpd/conf/httpd.conf'],
  }

}

Lets drive the point home with a couple diagrams. So, we have our site.pp files, which holds our node definitions, and we define our fw pre and post firewall rules, which I think of as a standard set of rules, that applies globally across all nodes, then as you include modules to each node, those get tallied up and each nodes get a custom iptables firewall. For example, one node might just have openssh included, but another might have openssh and a few others.

Puppet iptables flow

One really cool thing about this system, is that as each node checks in with the puppet master, these rules are evaluated, and updated as needed. So, if you had to add or update a rule to mitigate an attack, this could be handled across a large number of machines quickly.

Okay, enough theory, lets look at in practice.

I am using the Vagrant box from episode #8, same Vagrantfile, but I have just updated the puppet files to include the Puppet Labs Firewall module, and the scripts listed in the episode transcript. Lets fire the Vagrant box up, by running Vagrant up. The virtual machine booted and applied our puppet scripts, lets quickly review the output, before we connect to the machine. Up here, we have pretty standard output, then we see the puppet output, which talks about updating the message of the day file and tweaks to openssh, then we have puppet removing the default firewall rules, and then it starts applying our rules.

Lets connect to the Vagrant box and have a look, by running Vagrant ssh, then lets sudo to root. So, lets see what the rules look like by running iptables -L -n.

At a high level we have out three iptables chains, INPUT, FORWARD, and OUTPUT. Then we have some base rules for allowing icmp, localhost, and established or related traffic, and allowing incoming ssh connections, then we log and drop all other incoming traffic.

For the forward chain, we just log and drop the traffic. Since we have an outgoing firewall too, we needed to have some allow rules for, established or related traffic, dns queries, outgoing httpd, and pings, then we log and drop all other traffic. These logged packets will go to syslog by default, you can typically find them in /var/log/messages, but you probably would want to send these to a specific log file.

You will notice, up here in the input chain, we have our openssh rule, which is the result of including the openssh module, but there us nothing to include incoming httpd. Lets jump back to the node definition, and include httpd to see what happens. We can use these lines up here to re-run the puppet agent to update the node.

So, we are told the puppet agent to run again, using the updated node definition file, and it picks up that we need to install httpd, and the agent makes sure it is started, and along with the module we get our firewall rule. Lets verify that is the case by running iptables -L -n again. Looks like it is working, we have our httpd and openssh rules, one thing I love about this module is that it includes these comments in the iptables output, makes it really handy to see where these rules are coming from!

Lets try and remove the httpd module and see what happens. Lets go ahead and delete it, and then run puppet again. Notice that puppet picks up the change and removes the rule! The power this give you when dealing with hundreds or thousands of machines is pretty nice!

So, to bring this full circle, if you need to manage large clusters, and many one off machines, puppet can make managing iptables firewalls easy. You can also sleep safely knowing the rules are active and continually updated as your configuration evolves.