Getting a Let's Encrypt cert for pfSense web UI

The “Problem”

It was annoying to keep telling my browser “yeah, yeah it’s OK to proceed to this unsafe site” whenever I wanted to access the web UI on my pfSense device. An annoyance class problem but still fun to solve for robust future laziness aka functional auto-renew!

I’ll cover the high level idea and some of the implementation detail I found clutch, but this is not at all meant to be an exhaustive end-to-end HowTo. That said, if you’re stuck where I was this might help you.

Ingredients and my version of them

• firewall hardware - Netgate 4100
• firewall software - pfSense 23.09.1-RELEASE (amd64)
• Let’s Encrypt client - pfsense package acme v0.7.5
• external, Internet resolvable DNS - AWS Route53
• LAN-hosted webserver - Apache2 on Ubuntu 22.04.4 LTS
• Firewall NAT rules

The Basic Idea

In order for Let’s Encrypt to issue you a certificate, its servers have to be able to access the device you want a certificate via HTTP. Since I don’t want to expose the web server on my firewall to the Internet, this presents a problem.

The solve was to have a LAN-based web host answer the Let’s Encrypt response to the pfSense acme client’s call via Webroot FTP (not actually FTP) method configured in Services > Acme certificates.

Making it work

First, understand that my solution lies at the intersection of my interest and laziness. There is likely a more secure, more whizbang way to do this. I stopped at “works, and seems reasonably secure”, where my sense of ‘secure’ is, as mentioned, somewhat subject to my laziness.

Next, let’s understand the Webroot FTP method of answering a Let’s Encrypt (“LE”) query.

1. The web server you want to issue a certificate to, runs a client that asks LE’s servers for a certificate
2. acme opens an SSH session to your on-LAN web server and puts the challenge response it generated there, to an absolute (not relative) directory you specify, that matches where it told LE to look for it relative to the webroot
3. LE’s servers do a lookup in public DNS for the name of the host you said you wanted a certificate for
4. LE’s servers make an HTTP request of that host, by name, for the file the acme client told it to look for (the challenge response file)
5. Configured robustly, this all happens again every 60 days without you having to think about it again

Stuff you need to set up

• A web server on your LAN aka lanwebhost
• Firewall rules that setup a NAT from your external, routable IP (aka externalIP) and ports 80to port 80 on the LAN interface of the web server aka
• A local account on the web server that can be SSHed to by password (I know, I know. Don’t expose port 22 on this host to the Interwebz, ok? Ok.), aka acmeuser
• Create external DNS host entries that specify the name of the host you want to configure, and if necessary, any intervening subdomain(s)
• An Acme account key in Services > Acme certificates > Account keys
• Configure the acme client to use Webroot FTP as the method for the challenge response

Examples of each

• Web server: Straightforward Apache2 setup on Ubuntu 22.04.4 LTS that - critically in my experience - has a VirtualHost config for host.yourdomain.tld (if subdomain in play, host.sudomain.yourdomain.tld), aka lanwebhost

  • lanwebhost also must allow SSH by password

• An externally routable IP, aka your ISP router’s public interface, aka externalIP

• pfSense rules that NAT your externalIP:80 to lanwebhost:80

• A role account (it’s only used for this one thing) on lanwebhost, aka acmeuser

  • a user group (/etc/groups) for this user, aka acmeusergroup
  • Don’t Forget: You need to set the permissions on the sftp target folder correctly for this user to be able to write files there, e.g. making the folder group-writeable by acmeuser’s group ID, e.g. chown acmeusergroup /www/default/htdocs/.well-known/acme-challenge/; chmod g+rwx /www/default/htdocs/.well-known/acme-challenge/

• Externally resolvable hostname for your externalIP that matches the name you put in the certificate request, aka host.yourdomain.com or host.bar.yourdomain.com if you have fancy internal subdomain.

  • If you use a subdomain, you’ll need your external DNS to have an entry for the subdomain and for the host in that subdomain. e.g subdomain bar.yourdomain.tld which resolves to externalIP, and host.bar.yourdomain.tld which also resolves to your externalIP
  • I use a dynamic DNS client on the pfSense to register externalIP in AWS’s Route53. Setting this up is beyond the scope of this guide; there are good guides out there, I used ’em!

• You can give your account key any name you want, for example moo

• Likewise, you can give your Acme config any name you want, for example, foo

• The configuration details of the certificate issuing setup is where names start to matter.

  • Domainname must be the externally resolvable hostname you want a certificate issued in the name of / for
  • Server has to be the addressing by name or IP that your pfSense device can access your lanwebhost at, e.g. labwebhost.lan or 192.168.0.2 or whatever
  • Username has to be the name of acmeuser account created for this
    • for maximum laziness and less security, you could use your personal account
    • whatever account, see Don’t Forget: above and set folder perms so this account can write to the sftp target folder
  • Password probably you can guess has to be the password for acmeuser
  • Folder is the absolute path name - the full path name starting from the root directory , / on your lanwebhost - where the acme client will SSH the challenge response to, aka sftp target folder

Troubleshooting

• pfSense will give you UI output that directs you to a log file if things don’t go right. There are good clues there.
• On lanwebhost looking at the access log will help determine if your pfSense host is able to ssh into it ; sudo tail -f /var/log/auth.log
• On lanwebhost, looking at the HTTP server access logs helps a bunch; sudo tail -f /var/log/apache2/access.log for Default and sudo tail -f /var/log/apache2/other_vhosts_access.log for VirtualHosts
  • This was the most clutch for me, reminding me that I needed a VirtualHost entry on the server that matched the hostname Let’s Encrypt was trying to access. Without this lanwebhost was defaulting to the default host config, which, even though the absolute path to the challenge response file was the same for both default host and the VirtualHost host.bar.youdomain.tld, the default server would not respond with the file. Once there was a properly named VirtualHost, life was good.

Hopefully at this point, your browser has stopped complaining, and there’s no more of this kind of thing, below!