I recently installed Fail2Ban on my personal mail/web host as the number of "bad actors" has climbed a lot in recent years and I no longer felt comfortable just allowing them to pummel my server.
I'm writing up another blog post shortly on everything I did and how it all works; however, I had one issue that kept cropping up:
fail2ban-client status recidive returned the following:
Status for the jail: recidive |- Filter | |- Currently failed: 158 | |- Total failed: 741 | `- File list: /var/log/fail2ban.log `- Actions |- Currently banned: 8 |- Total banned: 8 `- Banned IP list: 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168
So, I should have 8 IPs banned in my firewall, right? Well, running
ufw status | grep REJECT returned:
Anywhere REJECT 22.214.171.124 Anywhere REJECT 126.96.36.199 Anywhere REJECT 188.8.131.52 Anywhere REJECT 184.108.40.206
Only half of the expected IPs are actually being blocked. Just to make sure it wasn't an issue with UFW, I checked the IP-Chains as well:
# iptables --list | grep REJECT REJECT all -- polishlid.com anywhere reject-with icmp-port-unreachable REJECT all -- 220.127.116.11 anywhere reject-with icmp-port-unreachable REJECT all -- relay.cropkingchemicals.com anywhere reject-with icmp-port-unreachable REJECT all -- 18.104.22.168 anywhere reject-with icmp-port-unreachable
Yep, same 4 hosts being rejected. So, the issue was with Fail2Ban.
I then started combing around looking for the issue, when I stumbled upon this thread: Bug 1379110 - IP's show as banned, but aren't. Essentially, Fail2Ban versions prior to v0.9.5 have a bug when interacting with ip-chains whereby following a shutdown of the server, the "--match-set fail2ban-sshd" rules are not being removed. So, when you first start the server back up, or restart it, those rules already exist and are not re-created with the contents of the Jail in question...you'll get new ones, but previous ones will be lost.
So, I checked the version of fail2ban in Ubuntu 16.04 LTS:
# fail2ban-server -V Fail2Ban v0.9.3
I was most likely experiencing the same bug. However, I was current with all of my updates on my Ubuntu server. I was now faced with figuring out how to upgrade just the Fail2Ban package from Ubuntu 16.10 (Yakkety Yak) without upgrading my whole system...
APT Package Pinning to the rescue
Here are the steps to upgrade only Fail2Ban (or any other package you want) to the latest version, without upgrading anything else.
APT Package Pinning
APT provides the ability to very precisely control the packages you're installing in your Linux distribution (one of the advantages that APT has over YUM). You can read up in detail on this by doing a
man apt preferences.
To pin the Fail2Ban package:
/etc/apt/sources.list.d/yakkety.listand replace all occurrences of "xenial" with "yakkety". -- This will be considered a new APT source and you'll pull all of the relevant indexes for this release (Ubuntu 16.10).
- Create the file
/etc/apt/preferences.d/yakketypinswith the following lines:
Package: fail2ban Pin: release n=yakkety Pin-Priority: 990 Explanation: Uninstall or do not install any Ubuntu-originated Explanation: package versions other than those in the xenial release Package: * Pin: release n=xenial Pin-Priority: 900 Package: * Pin: release o=Ubuntu Pin-Priority: -10
Essentially, you're first pinning Fail2Ban to a specific release (which just happens to be a future release from where you are). You then pin EVERYTHING else to the release you're actually on. Thus, only the packages pinned to a specific release will actually get upgraded.
Now perform the upgrade:
apt-get update-- you should see this update all of the repositories for both xenial and yakkety.
Your upgrade should just reveal the package you pinned, plus any dependencies:
# apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: fail2ban libarchive13 2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. # fail2ban-server -V Fail2Ban v0.9.5
Outstanding! Now I have an upgraded version of Fail2Ban. Let's see if it fixed the problem:
# fail2ban-client status recidive Status for the jail: recidive |- Filter | |- Currently failed: 149 | |- Total failed: 186 | `- File list: /var/log/fail2ban.log `- Actions |- Currently banned: 10 |- Total banned: 10 `- Banned IP list: 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206
Well, now I have 10 hosts revealed, but what about UFW?
# ufw status | grep REJECT Anywhere REJECT 220.127.116.11 Anywhere REJECT 18.104.22.168 Anywhere REJECT 22.214.171.124 Anywhere REJECT 126.96.36.199 Anywhere REJECT 188.8.131.52 Anywhere REJECT 184.108.40.206 Anywhere REJECT 220.127.116.11 Anywhere REJECT 18.104.22.168 Anywhere REJECT 22.214.171.124 Anywhere REJECT 126.96.36.199
All of them are present in the firewall and being actively rejected!
Note: since you've pinned essentially ALL of your packages to either Xenial Xerus or Yakkety Yak, you will probably have to remove this preference in order to perform a "dist-upgrade" in the future...just something to keep in mind.