Tag Archives: iptables

More Puppet Stuff

3 Replies

I’ve been continuing to migrate to my new server setup and Puppetising along the way, the outcome is yet more Puppet modules:

  1. The puppetlabs-firewall module performs very poorly with large rulesets, to work around this with my geoip/rirs module, I’ve gone and written puppet-speedychains, which generates iptables chains outside of the one-rule, one-resource Puppet logic. This allows me to do thousands of results in a matter of seconds vs hours using the standard module.
  2. If you’re doing Puppet for any more than a couple of users and systems, at some point you’ll want to write a user module that takes advantage of virtual users to make it easy to select which systems should have a specific user account on it. I’ve open sourced my (very basic) virtual user module as a handy reference point, including examples on how to use Hiera to store the user information.

Additionally, I’ve been working on Pupistry lightly, including building a version that runs on the ancient Ruby 1.8.7 versions found on RHEL/CentOS 5 & 6. You can check out this version in the legacy branch currently.

I’m undecided about whether or not I merge this into the main branch, since although it works fine on newer Ruby versions, I’m not sure if it could limit me significantly in future or not, so it might be best to keep the legacy branch as special thing for ancient versions only.

First thoughts and tips on EL 7

3 Replies

Generally one RHEL/CentOS/Scientific Linux (aka EL) release isn’t radically different to another, however the introduction of EL 7 is a bit of a shake up, introducing systemd, which means new init system, new ways of reading logs plus dropping some older utilities you may rely on and introducing new defaults.

I’m going through and upgrading some of my machines currently so I’ve prepared a few tips for anyone familiar with EL 4/5/6 and getting started with the move to EL 7.

 

systemd

The big/scary change introduced by RHEL 7 is systemd – love it or hate it, either way it’s here to stay. The good news is that an existing RHEL admin can keep doing most of their old tricks and existing commands.

Red Hat’s “service” command still works and now hooks into either legacy init scripts or new systemd processes. And rather than forcing everyone to use the new binary logging format, RHEL 7 logs messages to both the traditional syslog plain text files, as well as the new binary log format that you can access via journalctl – so your existing scripts or grep recipes will work as expected.

Rather than write up a whole bunch about systemd, I recommend you check out this blog post by CertDepot which details some of the commands you’ll want to get familiar with. The Fedora wiki is also useful and details stuff like enabling/disabling services at startup time.

I found the transition pretty easy and some of the new tricks like better integration between output logs and init are nice changes that should make Linux easier to work with for new users longer term thanks to better visibility into what’s going on.

 

Packages to Install

The EL minimum install lacks a few packages that I’d consider key, you may also want to install them as part of your base installs:

  • vim-enhanced – No idea why this doesn’t ship as part of minimum install so as a vim user, it’s very, very frustrating not having it.
  • net-tools – this provides the traditional ifconfig/route/netstat family of network tools. Whilst EL has taken the path of trying to force people onto the newer iproute tools there are still times you may want the older tools, such as for running older shell scripts that haven’t been updated yet.
  • bind-utils – Like tools like host or nslookup? You’ll want this package.
  • mailx – Provides the handy mail command for when you’re debugging your outbound mail.

 

Networking

Firstly be aware that your devices might no longer be simple named ethX, as devices are now named based on their type and role. Generally this is an improvement, since the names should line up more with the hardware on big systems for easier identification, and you can still change the device names if you prefer something else.

Changing the hostname will cause some confusion for long time RHEL users, rather than a line in /etc/sysconfig/network, the hostname is now configured in /etc/hostname like other distributions.

The EL 7 minimum installation now includes NetworkManager as standard. Whilst I think NetworkManager is a fantastic application, it doesn’t really have any place on my servers where I tend to have statically configured addresses and sometimes a few static routes or other trickiness like bridges and tunnels.

You can disable network manager (and instead use the static “network” service) by running the following commands:

systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl restart network

Red Hat have documentation on doing static network configuration, although it is unfortunately weak on the IPv6 front.

Most stuff is the same as older versions, but the approach of configuring static routes bit me. On EL 5 you configured a /etc/sysconfig/network-scripts/route-ethX file to define IPv4 and IPv6 routes that should be created when that interface comes up. With EL7 you now need to split the IPv4 and IPv6 routes apart, otherwise you just get a weird error when you bring the interface up.

For example, previously on an EL 5 system I would have had something like:

# cat /etc/sysconfig/network-scripts/route-eth1
10.8.0.0/16 via 10.8.5.2 dev eth1
2001:db8:1::/48 via 2001:db8:5::2 dev eth1
#

Whereas you now need something like this:

# cat /etc/sysconfig/network-scripts/route-eth1
10.8.0.0/16 via 10.8.5.2 dev eth1
#

# cat /etc/sysconfig/network-scripts/route6-eth1
2001:db8:1::/48 via 2001:db8:5::2 dev eth1
#

Hopefully your environment is not creative enough to need static routes around the place, but hey, someone out there might always be as crazy as me.

 

Firewalls

EL 7 introduces FirewallD as the default firewall application – it offers some interesting sounding features for systems that frequently change networks such as mobile users, however I’m personally quite happy and familiar with iptables rulesets for my server systems which don’t ever change networks.

Fortunately the traditional raw iptables approach is still available, Red Hat dragged their existing iptables/ip6tables service scripts over into systemd, so you can still save your firewall rules into /etc/sysconfig/iptables and /etc/sysconfig/iptables respectively.

# Disable firewalld:
systemctl disable firewalld
systemctl stop firewalld

# Install iptables
yum install iptables-service
systemctl enable iptables
systemctl enable ip6tables
systemctl start iptables
systemctl start ip6tables

 

LAMP Stack

  • Apache has been upgraded from 2.2 to 2.4. Generally things are mostly the same, but some modules have been removed which might break some of your configuration if you take a lift+shift approach.
  • MySQL has been replaced by MariaDB (community developed fork) which means the package names and service have changed, however all the mysql command line tools still exist and work fine.
  • PHP has been upgraded to 5.4.16 which a little bit dated already – over the lifespan of EL 7 it’s going to feel very dated very quickly, so I hope Red Hat puts out some php55 or php56 packages in future releases for those whom want to take advantage of the latest features.

 

Other Resources

  1. If you haven’t already, check out Red Hat’s release notes,they detail heaps of new features and additions to the platform.
  2. To learn more about the changes from previous releases, check out Red Hat’s Migration Guide as a starter.
  3. My guide to running EL 7 on EL 5 as a Xen guest for those of you running older Xen hypervisors.

ip6tables: ipv6-icmp vs icmp

7 Replies

I run a fully dual stacked IPv6+IPv4 network on my servers, VPNs and home network – part of this is that I get to discover interesting new first-adopter pains with living in the future (like Networkmanager/Kernel bugs, Munin being stupid, CIFS being failtastic and providers still stuck in the IPv4 only 1980s).

My laptop was experiencing frustrating issues where it was unable to load content from some IPv6 enabled website providers. In my specific case, I was having lots of issues with page loads from WordPress and Gravatar timing out when connecting to them via IPv6, but no issues when using IPv4.

I noticed that I was still able to ping6 the domains in question and telnet to port 80 successfully, which eliminates basic connectivity issues from being the cause. Issues like this where connectivity tests succeed, but actual connections fail, can be a symptom of MTU discovery issues which are a particularly annoying networking glitch to experience.

If you’re behind a WAN link such as ADSL, you’re particularly likely to be affected since ADSL and PPP overheads drop the size of the packets which can be used – in my case, I can only send a maximum of 1460 byte packets, whereas the ethernet default that my laptop will use is 1500 bytes.

In a properly functioning network, your computer will try and send 1500 byte packets to the internet, but the router which has the 1460 byte uplink to your ISP will refuse the packet and advise your computer that this packet is too large and that it needs to break it into smaller ones and try again. This happens transparently and is a standard feature of networking.

In a fucked up improperly functioning network, your computer will try and send the 1500 byte packet to the internet, but no notification advising the correct MTU size is returned or received. In this case your computer keeps trying to re-send the packet until a timeout occurs – from your computer’s perspective, the remote host is unreachable.

This MTU notification is performed by the ICMP protocol, which is more commonly but incorrectly known as being “ping” [whilst ping is one of the functions performed by ICMP, there are many other it’s responsible for, including MTU discovery and connection refused messages].

It’s not uncommon for MTU to be broken – I’ve seen too many system and network administrators block ICMP entirely in their firewalls “for security”, not realising that there’s a lot in ICMP that’s needed for proper operation of a network. What makes the problem particularly bad, is that it’s inconsistent and won’t necessarily impact all users, which leads to those administrators disregarding it as not being an issue with their infrastructure and even blaming the user.

Sometimes the breakage might not even be in a network you or the remote endpoint control – if there’s a router somewhere between you and the website you’re trying to access which has a smaller MTU size and blocks ICMP, you may never receive an MTU notification and you lose the ability to connect to the remote site.

At other times, the issue might be more embarrassing – is your computer itself refusing the helpful MTU notifications being supplied to it by the routers/systems it’s attempting to talk with?

I’m pretty comfortable with iptables and ip6tables, Linux’s IPv4 and IPv6 firewall implementations and use them for locking down servers, laptops as well as conducting all sorts of funky hacks that would horrify even the most bitter drugged up sysadmin.

However even I still make mistakes from time to time – and in my case, I had made a big mistake with the ICMP firewalling configuration that made me the architect of my own misfortune.

On my laptop, my IPv4 firewall looks something like below:

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
  • We want to trust anything from ourselves (duh) with -i lo -j ACCEPT.
  • We allow any established/related packets being sent in response to whatever connections have been established by the laptop, such as returned traffic for an HTTP connection – failure to define that will lead to a very unhappy internet experience.
  • We trust all ICMP traffic – if you want to be pedantic you can block select traffic, or limit the rate you receive it to avoid flood attacks, but a flood attack on Ethernet against my laptop isn’t going to be particularly effective for anyone.
  • Finally refuse any unknown incoming traffic and send an ICMP response so the sender knows it’s being refused, rather than just dropped.

My IPv6 firewall looked very similar:

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p icmp -j ACCEPT
ip6tables -A INPUT -j REJECT --reject-with icmp6-adm-prohibited

It’s effectively exactly the same as the IPv4 one, with some differences to reflect various differences in nature between IPv4 and IPv6, such as ICMP reject options. But there’s one horrible, horrible error with this ruleset…

ip6tables -A INPUT -p icmp -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp -j ACCEPT

Both of these are valid, accepted ip6tables commands. However only -p ipv6-icmp correctly accepts IPv6 ICMP traffic. Whilst ip6tables happily accepts -p icmp, it doesn’t effectively do anything for IPv6 traffic and is in effect a dud statement.

By having this dud statement in my firewall, from the OS perspective my firewall looked more like:

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -j REJECT --reject-with icmp6-adm-prohibited

And all of a sudden there’s a horrible realisation that the firewall will drop ALL inbound ICMP, leaving my laptop unable to receive many important messages such as MTU and rejected connection notifications.

By correcting my ICMP rule to use -p ipv6-icmp, I instantly fixed my MTU issues since my laptop was no-longer ignoring the MTU notifications. :-)

My initial thought was that this would be horrible bug in ip6tables, surely it should raise some warning/error if an administrator tries to use icmp vs ipv6-icmp. The man page states:

 -p, --protocol [!] protocol
    The  protocol of the rule or of the packet to check.  The speci-
    fied protocol can be one of tcp, udp, ipv6-icmp|icmpv6, or  all,
    or  it  can be a numeric value, representing one of these proto-
    cols or a different one.

So why is it accepting -p icmp then? Clearly that’s a mistake, it’s not in the list of accepted protocols…. but further reading of the man page also states that:

A protocol name from /etc/protocols is also allowed.

Hmmmmmmm…..

$ cat /etc/protocols  | grep icmp
icmp       1    ICMP         # internet control message protocol
ipv6-icmp 58    IPv6-ICMP    # ICMP for IPv6

Since /etc/protocols defines both icmp and ipv6-icmp as being known protocols by the Linux OS, ip6tables accepts the protocol argument of icmp without complaint, even though the kernel effectively will never be able to do anything useful with it.

In some respects it’s still a bug, ip6tables shouldn’t be letting users select protocols that it knows are wrong, but at the same time it’s not a bug, since icmp is a valid protocol that the kernel understands, it’s just that it simply will never encounter it on IPv6.

It’s a total newbie mistake on my part, what makes it more embarrassing is that I managed to avoid making this mistake on my server firewall configurations yet ended up doing it on my own laptop. Yet it’s very easy to do, hence this blog post in the hope that someone else doesn’t get caught with this in future.