Recreational frustration in home systems administration: Netgate pfSense and ssh blocking. Default deny, denied?!

Preface

Upon every home systems administration hobbyist’s life, some firewall upgrade woes must fall. It’s a very foot-rich target environment, to mix my metaphors.

This is not meant to be a detailed walkthrough. It is an anecdotal tale of the upgrade experience which may be of value to you, Dear Reader, in terms of highlighting some lessons learned which you may subsequently vicariously avoid. Also, I found a behavior that I question whether is working as intended (WAI). I may file a case, or search the forum or something (no dice. hrmph. ).

Upgrade Installation

I use a Netgate 4100 running pfSense Plus which I just last weekend upgraded to 23.09.1 . As usual, and you’d think I’d learn by now, I did not create (nor use ) a step-by-step plan to do the upgrade. Instead, at least having learned that I need to keep a USB drive with the release image on it handy, I went to my console and got started.

Very soon I was looking at a non-bootable OS and no hope of performing the “Recover config.xml ” method. Well. That’s what the imaged USB drive is for! Imaging the 4100 I proceeded to recover my config.xml from my backup. At least I had that!

Installation Quirk: Next, I had remind myself that in order to get a web-ui available, I needed to move one of the interfaces off the default RFC-1918 address to Anything Else (or maybe just re-config it in situ? i unno), and turn on DHCP on that interface, before a client (my laptop, with the backup config.xml on it) will connect. This may be WAI, since pfSense hasn’t been told anything about interfaces. But, pfSense does (and did, in my case) auto-detect connections and with the WAN port connected to my ISP and some of the OPT ports connected to client devices, you’d think maybe I’d be able to talk to it right away? Anyway. Config recovered!

But, uh, this is when I notice that I had flashed the 4100 not to 23.09.1, but to 23.05.1. ‘Cause that’d come out when I started to think about updating, so I’d gotten the image for that, and then later the newer image, and I was sure I’d imaged the USB with the latest one yep yep nope sigh.

So I ran the upgrade through the UI after all, and that went fine.

And that, Dear Reader, is when I found the behavior I think is not WAI. But if it is…why?

WAI? Why?!

Long Ago, back when I was still running community edition pfSense, I left an open port 22 for remote SSH access to my stack (look I have a few computers, ok?) all the time. Then I started looking at the logs and reminded myself that’s a bad idea. You can spend a lot of monthly bandwidth quota on assholes probing your open ports if you know what I mean (Mom. Hi. This is not dirty. But it’s definitely innuendo.)

So I decided to stop doing that, and - and this is the crucial point - removed the rule allowing incoming 22 and let the default deny rule block that port. Cool. Then I bought a new Netgate 4100 (order history says May'22) and was moved in that purchase to pfSense Plus, which comes with such a purchase. I don’t think I cared much at the time, but OK, sure.

Fast foward to this weekend. I just updated re-installed so I’m spending some time poking around in the guts of the config and the logs because that’s fun. Wait. Why is sshguard making so many log entries?

• port 22 ssh is blocked by default deny, right?
• I definitely do not have a rule allowing port 22 ssh
• I have pfBlockerNG configured to only allow traffic that originates from some geographical regions and the source is not on any denylist subscriptions
• pfSense is definitely configured to allow 22 ssh with an allowlisted subnet (my local management network) that bypasses sshguard protections

OK. But I think that those last two bullet points are the issue. This appears to setup SSH access from the world not just the trusted subnet. Well, fine in terms of “doing what you told it to do” re: only the trusted subnet bypasses sshguard’s attack protections. But really, should it be WAI that w/o a rule to allow it, the port is now open to the world?! And apparently this has been True For A Long Time On My Network (since… May'22 maybe?!) And also, the pfBlockerNG geo allow rule seems to allow access that then allows SSH access via the sshguard setup regardless of rules written directly on the ISP interface.

Doesn’t seem like that ought to be true. Feels like it ought to open 22 to the specified trusted net(s) with an explicit rule like the anti-lockout rule . But it doesn’t. There’s no rule for this access visible in the webui (no, I haven’t checked the rule list via CLI).

So, how do I stop it?

Corking the SSH hole

I’ll save you a sordid tale of my thrashing around, remembering behaviors of firewalls that I no longer touch often enough that they are still muscle memory, and distill it down to these points:

• I added floating Match rules to watch for any 22 traffic
• Noting which interfaces addresses get that traffic from what sources, I wrote floating rules for the ISP interfaces (I have two ISP interfaces, For Reasons (that Reason is all consumer grade links are lossy and I work from home and it stuck in my craw that I might have to drive to the office because my !@#!#!@# ISP went down, you know?))
• These rules need to be set to quick so that the pfBlockerNG rules that follow don’t subsequently allow the 22 traffic
• By default, pfBlockerNG makes Floating Rules that are quick and these will move around based on a setting in Firewall > pfBlockerNG > IP which keeps putting them above - and thus processesed before - the 22 block quick floating rules. Not helpful for my use case. Well, for one of them. Definitley pfBlockerNG is helpful!
• You can stop this by unchecking the following setting in Firewall > pfBlockerNG > IP

Whew!

OK. Hope that was a) fun if you are of such technerd inclination b) useful

P.S.

I found (by using a VPN to get outside my firewall and nmaping back to home) that port 53 for unbound (DNS Resolver service) was also open to the Internet without an explicit rule anywhere to allow this!!! I fixed it with the same process above for that port.