Rocky Server Stack Deep Dive: 2023 Part 3.1

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project on my home server. Let’s get started!

Not My Problem

In the greater context of setting up Pi-Hole (network ad blocker) on my home server, ButtonMash, I’ve learned a thing or two about how the Domain Name Service (DNS) works and what happens when I break it locally. Normally, when a device connects to a network, a DHCP server (often on the router) advertises a DNS server. When resolving a URL, this DNS server either has the answer cached or ask another DNS server until a match is found or the system gives up.

My Internet Service Provider (ISP) has been having some trouble with its site (including our main e-mail). Not once, but twice my family asked if I was doing something with the Internet. Both times, I used a terminal application called “traceroute” to display [most of] the hops my requests went. Ones handled by ButtonMash were very short – (I tested buttonmash.lan and a known entry on my blocklist), while others took up to 30 hops. My ISP’s site fell in the later category.

However: one cell phone in the family was still reaching our ISP’s site while on cell data. This meant that the site was fine, but the failure was with the larger DNS system (or most of their servers were down behind their reverse proxy, but I thought of that later). In either case, I looked at a couple “Is it down?” type sites, and concluded the outage was most certainly not my problem.

Unbound

But I had a project to try. Unbound is a tool for increasing digital privacy by setting up a recursive DNS server. Every domain is registered at its authoritative DNS server. When Unbound receives a request, it finds the domain’s authoritative DNS server and caches it for later. This reduces digital footprints in DNS server logs, making you harder to track and reducing your vulnerability to a hacked/confused DNS servers.

I’ve been interested in building my own Unbound OCI “Docker” container for a while as there’s no officially maintained one, but I went ahead and downloaded an image from docker.io based on the quality of documentation. I spun up a container in Podman and pointed Pi-Hole at it. It worked first try with no special configuration.

It just so happened that when I brought the fix online, we were on a call with tech support, and I was able to pass my diagnosis back to our ISP to help them restore service to their customers in my community – however large or small that number may be.

Takeaway

What’s with no persistence volumes on this container? If it resets, it will have to start over on caching. If/when I come back in a few months, I may take a closer look. Otherwise, this has been a big shortcut I can live with.

Final Question

Have you worked with Unbound before? Would it even benefit from a persistence volume?

I look forward to hearing from you on my Socials!

My PiHole is “Half Baked”

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am installing PiHole. With luck, I’ll have be configuring some of its other functions to augment my home’s network as well. Let’s get started!

PiHole, Take II

I can rant about the evils of Google ‘til boredom do its part. However, this search engine is between inconvenient and impossible to ignore, given its impressive list of “hobbies” from STEM projects to smartphones. It’s an open secret few care to think about that their empire is built off user exploitation. I installed ad blocker browser plugins over their aggression last presidential election cycle.

Earlier this month, I read about Manifest v3, the new browser-plugin interface library created by Google. Their precautions against spyware just so happen to cripple ad blockers, among other legitimate plugins. This walking conflict of interest is set to roll out January, 2023, and Firefox is going along with it.

When a browser loads a web page, it asks a DNS service to translate the page’s URL into an IP address. It then finds, loads, and renders the page at that IP. This may involve loading other pages –such as ads– as elements of the original page. Network ad blockers protect you by fudging bad URL’s addresses.

Objectives

My main goal this week is to kill ads across my home network. Follow-up objectives include advanced PiHole features and a private DNS for even better protection.

Night 1

My first attempt at PiHole was messy. I set up PiHole OCI/“Docker” containers across my two servers – ButtonMash and my old laptop. Like before, the main router skipped IP’s on me. I had it repaired within an hour thanks to my same laptop functioning as a workstation with a static IP. With the router upgrade to my upstairs workstation, I easily archived its settings and outfitted it with its own wider network static IP – complete with a netmask wide enough to chase down its rogue counterpart should it shift again (Did I have laptop’s static IP netmask configured incorrectly this whole time?!).

Surprise! The expanded subnet didn’t work because the rogue router had its own subnet mask I was outside of. The dance was too involved for a play by play, but I only really felt helpless while trying to avoid hiking around to different workstations to clean up after this failed networking spell. As I reassembled the router for normal operation, I reasoned out that my router’s firmware is hardwired not to consider a DNS coming from a LAN connection, like I’m trying to do.

Flashing open source firmware is out of the question. For one, I wouldn’t know how to fix it and don’t have a replacement. Two: apparently its chipset manufacturer isn’t a fan of open source – the help thread I spotted recommended contacting OP’s government representative if he wanted to do anything about it.

Night 2

I did a bit of research before dismantling the network again. DHCP settings include optional fields for DNS requests. This should let me direct computers straight to PiHole instead of relaying the request in a convoluted workaround involving a NAT table and possibly causing a network loop.

This means each router is now a separate task. The responsible thing to do now is ensure my subnet router can behave before working on the main one. It’s not long before I fry my DNS settings. Navigation around my local network remains unaffected, but I eventually resort to restoring my backup from yesterday, re-applying the static IP, and updating the backup.

My best bet from here is to finalize my PiHole install. My initial container creation was the absolute minimum: port 80 web interface, port 53/TCP+UDP. There’s a lengthy list of environment variables to browse.

A Few Days Later

Jackpot! My mind cleared enough before bed to skim PiHole Docker’s documentation on GitHub. It has a list of example deployments – including a shell script. I converted it for Podman, entered my environment variables, and –during debugging– axed the logic for relaying logs as it was causing problems and I can view them directly with Cockpit-Podman.

PiHole User

But where to land it? I’ll eventually integrate as I master Caddy. Leaving the container running as root lets it use the proper ports, but I know better. Thanks to discoveries I spun off into last week’s project, I can now make more underprivileged, Cockpit-enabled users than I will ever need by using loopback the address (127.0.0.1/8).

The run script was easy to copy over to my new PiHole user. I gave it the directories it wanted as mountable volumes and shifted ports around until I was happy. I took the time to tidy up my firewall, combining a couple related entries and reclosing the normal DNS port.

I remember having issues with Vaultwarden’s stability over the course of days to weeks. The problem was occasionally annoying as Bitwarden only requires its home server when modifying the password vault, but PiHole will be sorely missed the moment it goes down. The one place I found the solution was in the official Podman troubleshooting guide on their GitHub [1]:

loginctl allow-linger userName

I sadly could not verify this was my previous, solution to my Vaultwarden long-term issues, but it’s not entirely unfamiliar, and it’s my best-informed guess.

DNS Port Forwarding

With PiHole secured in its own, easily accessible account, I soon experienced how picky DNS requests are about using the privileged port 53. All my attempts at manually telling OpenWRT to use port 5300 failed. I expect the the story will be the same if I try with on my main router.

I found the solution where Woody from b-woody.com blogged about almost the exact same project last May [2]: port forward port 53 to port 5300. Paranoid about goofing my firewall over command line I ran my version of Woody’s commands past r/TechSupport’s Discord channel. Moderator Donjuanal confirmed my omission of a trailing “:toaddr=”, but questioned my blind use of tcp, explaining how DNS clients default to udp for speed.

sudo firewall-cmd --zone=public --add-forward-port=port=53:proto=udp:toport=5300 --permanent

Even with this measure in place, I had to access the web console and tick Settings>DNS>Interface settings>Potentially dangerous options>Permit all origins before my local requests made it through. This may need to be addressed later.

Takeaway

I am so glad to have PiHole installed, even if it doesn’t appear to be doing much more than the uBlock Origin Firefox plugin. I’m researching the next segment though, and I estimate another week or more worth of work before it is configured alongside a private DNS server. Worth noting is that Firefox is leaving in the features ad block requires, despite potential security concerns. This is as good enough stopping point.

Final Question

Do you use PiHole? I’d be happy to hear about your experience.

I look forward hearing your answers on in the comments below or on my Socials.

Works Cited

[1] eriksjolund, “Podman\ Troubleshooting\ A list of common issues and solutions for Podman,” github.com, Nov. 19, 2022. [Online]. Available: https://github.com/containers/podman/blob/main/troubleshooting.md [Accessed Jan. 30, 2023].

[2] Woody, “Run PiHole in a rootless Podman container,” b-woody.com, May 12, 2022.[Online]. Available: https://b-woody.com/posts/2022-05-12-pihole-on-a-rootless-podman-container/ [Accessed Jan. 30, 2023].

[3] Can You Block It, “CAN YOU BLOCK IT?\ AN SIMPLE AD BLOCK TESTER” canyoublokit.com, 2021. [Online]. Available: https://canyoublockit.com/ [Accessed Jan. 30, 2023].