Solving Fail2Ban not banning IPs on Ubuntu 16.04

I recently installed Fail2Ban on my personal mail/web host as the number of "bad actors" has climbed a lot in recent years and I no longer felt comfortable just allowing them to pummel my server.

I'm writing up another blog post shortly on everything I did and how it all works; however, I had one issue that kept cropping up:

Running fail2ban-client status recidive returned the following:

Status for the jail: recidive  
|- Filter
|  |- Currently failed: 158
|  |- Total failed:     741
|  `- File list:        /var/log/fail2ban.log
`- Actions
   |- Currently banned: 8
   |- Total banned:     8
   `- Banned IP list:   185.127.19.155 203.87.129.135 80.82.77.203 94.102.51.31 85.194.111.39 23.254.215.195 46.21.147.118 164.132.62.70

So, I should have 8 IPs banned in my firewall, right? Well, running ufw status | grep REJECT returned:

Anywhere                   REJECT      94.102.51.31  
Anywhere                   REJECT      80.82.77.203  
Anywhere                   REJECT      203.87.129.135  
Anywhere                   REJECT      185.127.19.155  

Only half of the expected IPs are actually being blocked. Just to make sure it wasn't an issue with UFW, I checked the IP-Chains as well:

# iptables --list | grep REJECT
REJECT     all  --  polishlid.com        anywhere             reject-with icmp-port-unreachable  
REJECT     all  --  80.82.77.203         anywhere             reject-with icmp-port-unreachable  
REJECT     all  --  relay.cropkingchemicals.com  anywhere             reject-with icmp-port-unreachable  
REJECT     all  --  185.127.19.155       anywhere             reject-with icmp-port-unreachable  

Yep, same 4 hosts being rejected. So, the issue was with Fail2Ban.

I then started combing around looking for the issue, when I stumbled upon this thread: Bug 1379110 - IP's show as banned, but aren't. Essentially, Fail2Ban versions prior to v0.9.5 have a bug when interacting with ip-chains whereby following a shutdown of the server, the "--match-set fail2ban-sshd" rules are not being removed. So, when you first start the server back up, or restart it, those rules already exist and are not re-created with the contents of the Jail in question...you'll get new ones, but previous ones will be lost.

So, I checked the version of fail2ban in Ubuntu 16.04 LTS:

# fail2ban-server -V
Fail2Ban v0.9.3  

I was most likely experiencing the same bug. However, I was current with all of my updates on my Ubuntu server. I was now faced with figuring out how to upgrade just the Fail2Ban package from Ubuntu 16.10 (Yakkety Yak) without upgrading my whole system...

APT Package Pinning to the rescue

Here are the steps to upgrade only Fail2Ban (or any other package you want) to the latest version, without upgrading anything else.

APT Package Pinning

APT provides the ability to very precisely control the packages you're installing in your Linux distribution (one of the advantages that APT has over YUM). You can read up in detail on this by doing a man apt preferences.

To pin the Fail2Ban package:

  1. First cp /etc/apt/sources.list to /etc/apt/sources.list.d/yakkety.list
  2. Edit /etc/apt/sources.list.d/yakkety.list and replace all occurrences of "xenial" with "yakkety". -- This will be considered a new APT source and you'll pull all of the relevant indexes for this release (Ubuntu 16.10).
  3. Create the file /etc/apt/preferences.d/yakketypins with the following lines:
Package: fail2ban  
Pin: release n=yakkety  
Pin-Priority: 990

Explanation: Uninstall or do not install any Ubuntu-originated  
Explanation: package versions other than those in the xenial release  
Package: *  
Pin: release n=xenial  
Pin-Priority: 900

Package: *  
Pin: release o=Ubuntu  
Pin-Priority: -10  

Essentially, you're first pinning Fail2Ban to a specific release (which just happens to be a future release from where you are). You then pin EVERYTHING else to the release you're actually on. Thus, only the packages pinned to a specific release will actually get upgraded.

Now perform the upgrade:
1. apt-get update -- you should see this update all of the repositories for both xenial and yakkety.
2. apt-get upgrade

Your upgrade should just reveal the package you pinned, plus any dependencies:

# apt-get upgrade                       
Reading package lists... Done  
Building dependency tree  
Reading state information... Done  
Calculating upgrade... Done  
The following packages will be upgraded:  
  fail2ban libarchive13
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# fail2ban-server -V
Fail2Ban v0.9.5  

Outstanding! Now I have an upgraded version of Fail2Ban. Let's see if it fixed the problem:

# fail2ban-client status recidive
Status for the jail: recidive  
|- Filter
|  |- Currently failed: 149
|  |- Total failed:     186
|  `- File list:        /var/log/fail2ban.log
`- Actions
   |- Currently banned: 10
   |- Total banned:     10
   `- Banned IP list:   164.132.62.70 185.127.19.155 203.87.129.135 23.254.215.195 46.21.147.118 80.82.77.203 85.194.111.39 94.102.51.31 74.121.49.191 46.148.27.14

Well, now I have 10 hosts revealed, but what about UFW?

# ufw status | grep REJECT
Anywhere                   REJECT      94.102.51.31  
Anywhere                   REJECT      85.194.111.39  
Anywhere                   REJECT      80.82.77.203  
Anywhere                   REJECT      46.21.147.118  
Anywhere                   REJECT      23.254.215.195  
Anywhere                   REJECT      203.87.129.135  
Anywhere                   REJECT      185.127.19.155  
Anywhere                   REJECT      185.114.142.131  
Anywhere                   REJECT      119.152.138.0  
Anywhere                   REJECT      164.132.62.70  

All of them are present in the firewall and being actively rejected!

Problem solved.

Note: since you've pinned essentially ALL of your packages to either Xenial Xerus or Yakkety Yak, you will probably have to remove this preference in order to perform a "dist-upgrade" in the future...just something to keep in mind.

linux sysadmin
Tweet Post Share Update Email RSS

Leader, Mentor, Challenger, Educator, Network Engineer, System Administrator, Developer, Hacker, Writer, Diver, and Technology Explorer