I recently installed Fail2Ban on my personal mail/web host as the number of "bad actors" has climbed a lot in recent years and I no longer felt comfortable just allowing them to pummel my server.
I'm writing up another blog post shortly on everything I did and how it all works; however, I had one issue that kept cropping up:
Running fail2ban-client status recidive
returned the following:
Status for the jail: recidive
|- Filter
| |- Currently failed: 158
| |- Total failed: 741
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 8
|- Total banned: 8
`- Banned IP list: 185.127.19.155 203.87.129.135 80.82.77.203 94.102.51.31 85.194.111.39 23.254.215.195 46.21.147.118 164.132.62.70
So, I should have 8 IPs banned in my firewall, right? Well, running ufw status | grep REJECT
returned:
Anywhere REJECT 94.102.51.31
Anywhere REJECT 80.82.77.203
Anywhere REJECT 203.87.129.135
Anywhere REJECT 185.127.19.155
Only half of the expected IPs are actually being blocked. Just to make sure it wasn't an issue with UFW, I checked the IP-Chains as well:
# iptables --list | grep REJECT
REJECT all -- polishlid.com anywhere reject-with icmp-port-unreachable
REJECT all -- 80.82.77.203 anywhere reject-with icmp-port-unreachable
REJECT all -- relay.cropkingchemicals.com anywhere reject-with icmp-port-unreachable
REJECT all -- 185.127.19.155 anywhere reject-with icmp-port-unreachable
Yep, same 4 hosts being rejected. So, the issue was with Fail2Ban.
I then started combing around looking for the issue, when I stumbled upon this thread: Bug 1379110 - IP's show as banned, but aren't. Essentially, Fail2Ban versions prior to v0.9.5 have a bug when interacting with ip-chains whereby following a shutdown of the server, the "--match-set fail2ban-sshd" rules are not being removed. So, when you first start the server back up, or restart it, those rules already exist and are not re-created with the contents of the Jail in question...you'll get new ones, but previous ones will be lost.
So, I checked the version of fail2ban in Ubuntu 16.04 LTS:
# fail2ban-server -V
Fail2Ban v0.9.3
I was most likely experiencing the same bug. However, I was current with all of my updates on my Ubuntu server. I was now faced with figuring out how to upgrade just the Fail2Ban package from Ubuntu 16.10 (Yakkety Yak) without upgrading my whole system...
APT Package Pinning to the rescue
Here are the steps to upgrade only Fail2Ban (or any other package you want) to the latest version, without upgrading anything else.
APT Package Pinning
APT provides the ability to very precisely control the packages you're installing in your Linux distribution (one of the advantages that APT has over YUM). You can read up in detail on this by doing a man apt preferences
.
To pin the Fail2Ban package:
- First
cp /etc/apt/sources.list
to/etc/apt/sources.list.d/yakkety.list
- Edit
/etc/apt/sources.list.d/yakkety.list
and replace all occurrences of "xenial" with "yakkety". -- This will be considered a new APT source and you'll pull all of the relevant indexes for this release (Ubuntu 16.10). - Create the file
/etc/apt/preferences.d/yakketypins
with the following lines:
Package: fail2ban
Pin: release n=yakkety
Pin-Priority: 990
Explanation: Uninstall or do not install any Ubuntu-originated
Explanation: package versions other than those in the xenial release
Package: *
Pin: release n=xenial
Pin-Priority: 900
Package: *
Pin: release o=Ubuntu
Pin-Priority: -10
Essentially, you're first pinning Fail2Ban to a specific release (which just happens to be a future release from where you are). You then pin EVERYTHING else to the release you're actually on. Thus, only the packages pinned to a specific release will actually get upgraded.
Now perform the upgrade:
apt-get update
-- you should see this update all of the repositories for both xenial and yakkety.apt-get upgrade
Your upgrade should just reveal the package you pinned, plus any dependencies:
# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
fail2ban libarchive13
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
# fail2ban-server -V
Fail2Ban v0.9.5
Outstanding! Now I have an upgraded version of Fail2Ban. Let's see if it fixed the problem:
# fail2ban-client status recidive
Status for the jail: recidive
|- Filter
| |- Currently failed: 149
| |- Total failed: 186
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 10
|- Total banned: 10
`- Banned IP list: 164.132.62.70 185.127.19.155 203.87.129.135 23.254.215.195 46.21.147.118 80.82.77.203 85.194.111.39 94.102.51.31 74.121.49.191 46.148.27.14
Well, now I have 10 hosts revealed, but what about UFW?
# ufw status | grep REJECT
Anywhere REJECT 94.102.51.31
Anywhere REJECT 85.194.111.39
Anywhere REJECT 80.82.77.203
Anywhere REJECT 46.21.147.118
Anywhere REJECT 23.254.215.195
Anywhere REJECT 203.87.129.135
Anywhere REJECT 185.127.19.155
Anywhere REJECT 185.114.142.131
Anywhere REJECT 119.152.138.0
Anywhere REJECT 164.132.62.70
All of them are present in the firewall and being actively rejected!
Problem solved.
Note: since you've pinned essentially ALL of your packages to either Xenial Xerus or Yakkety Yak, you will probably have to remove this preference in order to perform a "dist-upgrade" in the future...just something to keep in mind.