Tag: Networking

Posted on

Responsibility of the Network

Good Morning from my Robotics Lab! This is Shadow_8472, and today I have a doozy of a network week to cover. Let’s get started!

Meet the Computers

  • Cerberus – the main star today. It is our new hardware firewall running OPNsense
  • Red Router – a tp-link gaming Wi-Fi router fancied up beyond what it should have been
  • LAB – my homelab with a few servers
  • LAN – everything else connected via Wi-Fi or Ethernet

Network Implosion

It all started with revisiting a .lan domain. Cerberus’ extensive webUI left me with the hunch I’d need one machine in charge of DHCP assigning dynamic IP addresses. Red Router’s “operation mode” to work as an access point was hidden in literally the last menu to click through.

It was afternoon and no one would be using the network for the 30 seconds to 5 minutes I estimated switching the LAB and LAN Ethernet cables from Red Router over to Cerberus would take. Nope. No traffic made it through. DHCP mis-configuration? Cue a slow back and forth, bopping a setting from a workstation and trying a different physical configuration. Eventually, Cerberus ended up on my desk with Red Router talking directly to our ISP’s gateway/modem.

Order of events is a bit fuzzy from here, but when the Wi-Fi stopped working, I was without a good access to online answers. I worked the problem into the night. Around 1:00 AM, I knew I had done too much for a clean reversion. For two hours, I worked in loops hoping to spot something different. So much waiting! Cerberus would behave on my desk, then fail when redeployed. Worse: when I re-connected all the wires to Red Router, it started dancing between 192.168.0.1 and 192.168.1.1 every 45 seconds or so. I configured its IP manually, but gave up on Internet by morning at 3:00 AM, preparing to concede to my father’s suggestion earlier about hiring a professional to untangle my mess.

The Next Day

Newtork Loop. In my brain fog, I had Red Router talking to itself on a cable leftover from removing Cerberus for the night. With the house to myself for several hours, I alternated between bursts of intense diagnostics and mental processing. Somewhere in there, I rebooted the ISP’s modem.

Around noon, I realized the extra ports on Cerberus aren’t a switch as is Red Router’s default configuration, but were following firewall rules – which explained its behavior the previous day when I tried a computer from LAB without anything in-between. At around 3 PM, I got a Discord notification while mentally checked out, letting me know the network was back on.

6 PM on the second day: I situated my workstation in Cerberus’ LAN port and a Raspberry Pi in one I named LAN2. I’d previously copied firewall rules from LAN to LAN2 and LAB, but to no obvious effect – until I had the two computers ping each other. LAN2 failed as expected, but LAN’s ping was returned. I corrected the interfaces’ rules to allow them to reach out, and that was it.

Fallout

Without going into too much detail, a subnet shift like this is a major undertaking for networks with static IP servers on them. Not only do the network and computers need to be adjusted, but all traces of the old subnet need to be corrected. NFS clients needed to be told where the server was now, and the NFS server shares needed to be updated about what IP’s were allowed to mount them. I also still have Bitwarden to clients to update at my leisure.

Takeaway

OPNsense is a heavy weight in terms of configuration options. It has a learning curve compared to products simple enough to for Grandma and Grandpa to use. I may have solved my own emergency, but it may be wise to get someone looking at it professionally anyway to grade my work and give me some pointers on rootless Podman mounting NFS shares, or other long-term places where I’ve gotten stuck.

Final Question

I admit: networking is more fun than I gave it credit for before I knew basically anything. I still find it a bit taxing to mentally reach around my mental map, but I manage. How do you visualize networks?

Posted on

Unboxing: Hardware Firewall (Protectli Vault)

Good Morning from my Robotics Lab! This is Shadow_8472 and today I have on my desk between my keyboard and monitors a new Protectli Vault running OPNsense. Let’s get started.

After at least a couple years tentatively researching hardware firewalls, it’s here. Let me tell you: it’s both a relief and a bit of pressure. I’m glad I’m no longer starting from scratch over and over again, but now I feel time pressure to deploy it despite my parents’ assurance that it’s much better to go at a responsible pace. And unless you’re a full time network specialist, that pace is longer than a week.

My Current Network and Its Weaknesses

At present, my home network starts with a box owned and controlled by my service provider. This gateway feeds into a gaming router before going out to a couple switches and Wi-Fi. One of my desktops has OpenWRT on a Raspberry Pi 4. ButtonMash, my home server, runs Podman containers for Vaultwarden (Password vault storage) and PiHole (DNS ad blocking). We have a Network Attached Storage by the hostname of GoldenOakLibry. Everything minus a couple workstations has battery backup in case the lights go out.

And when the lights do go out, the first big flaw comes out. While the network closet may last several hours, Power-hungry ButtonMash and GoldenOakLibry chew through their shared battery in around half an hour before I added ButtonMash’s twin, Joystick, as a development platform. When ButtonMash goes down, the network loses DNS so we can’t resolve URL’s.

Additionally, I’d like to move to a non-default set of internal IP addresses, like 10.59.102.X instead of 10.0.0.X or 192.168.0.X. While computers getting automatic IP’s over DHCP will essentially take care of themselves, I have invested quite a bit of time into static IP’s on NFS (Network File System), and when I move GoldenOakLibry’s IP, I’ll need to adjust the automounts for all systems accessing it, and that’s just a pain. I want to learn how a home domain works.

I also have a number of network-related projects I’ve done research for, but burned out on before solving. From memory, here’s a checklist of partial/incomplete/need-to-redo projects:

  • Feline Observation Pi (First prototype tested, needs overhaul)
  • Website for family photo archive (Needs hardware firewall, rootless Podman/NFS, booru/wiki)
  • Nextcloud (Early prototype successful, needs rootless Podman/NFS before production)
  • Beowulf cluster (Early research)
  • Rootless Podman/NFS (Heard from a developer and solution may not exist [yet])
  • UPS battery monitoring/shutdown before power failure (Research phase)
  • Caddy (First prototype in production, needs overhaul)
  • Unbound (Incomplete prototype)
  • Reverse VPN [mobile traffic] (Need Hardware Firewall)
  • Podman systemctl –user (In production, but I cannot reproduce at will)
  • Domain/Domain Controller (Background research incomplete)

Keep in mind that the notes on each item suggesting a direction are just the direction I’m leaning in at the moment without reflecting the new hardware. Replacing GoldenOakLibry with a server beefy enough to handle running Podman would solve my current need for rootless Podman/NFS. I may find a replacement for Caddy that also works as a Domain Controller. Does Caddy even do that? Let me check… Inconclusive; probably not. I don’t know enough about what to look for in a Domain Controller besides the name. Most of my time focused on researching Demilitarized Zones.

Demilitarized Zone and Roadmapping

Originally, I had a goal of deploying this new firewall/router configured with a demilitarized zone network structure. With hardware in hand, I learned a lot! But as I learned, I realized I needed to learn that much more to do the job right. A DMZ is basically a low security area of your network for serving stuff over an untrusted network (usually the wide open Internet) while protecting your Local Area Network. Ideally your LAN would have a separate physical router in case the one servicing the DMZ is ever compromised, but a homelab environment should be a small enough target that branching off from a single hardened router should be fine. My trouble is that I can’t fully tell where to put what.

I already know I want to move PiHole, Unbound, and similar projects related to internet traffic, and other projects I want lasting a bit longer into power outages onto the new router. OPNsense is a distribution of BSD and not Linux, so I expect I will need to look into a Linux Virtual Machine if BSD-based containers aren’t available. The gaming router I’m using now will still be our Wi-Fi access point, but I’d prefer to retire it from DHCP duty.

ButtonMash and Joystick are my enigma. I had plans of clustering them, but I may need one in the DMZ and one on the LAN. GoldenOakLibry belongs on the LAN so far as I can tell – as do all workstations.

Takeaway

There will be more thought to give it another week. I went ahead and hooked it up in place, but it didn’t work despite how I had previously had it working between my upstairs workstation and its rPi router. I’ve reverted the setup to how it was before, and I’ll need to take a closer look and do some further testing.

Final Question

What was the last piece of tech you unboxed?

Posted on

Rocky Server Stack Deep Dive: 2023 Part 2

MAJOR progress! This week, I’ve finally cracked a snag that’s been holding me for two years. Read on as I start a deep dive into my Linux server stack.

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am continuing renovations on my home server, ButtonMash. Let’s get started!

The daily progress reports system worked out decently well for me last week, so I’ll keep with it for this series.

Caddy is an all-in-one piece of software for servers. My primary goal this week is to get it listening on port 44300 (the HTTPS port multiplied by 100 to get it out of privileged range) and forwarding vaultwarden.buttonmash.lan and bitwarden.buttonmash.lan to a port Vaultwarden, a Bitwarden client I use, is listening on over Buttonmash’s loopback (internal) network.

Tuesday Afternoon

From my Upstairs Workstation running EndeavourOS, I started off with a system upgrade and reboot while my workspace was clean. From last week, I remember Vaultwarden was already rigged to have port 44300, but straight away, I remembered its preferred configuration is HTTP coming into the container, so I’ll be sending it to 8000 instead.

My first step was to stop the systemd service I’d set up for it and start a container without the extra Podman volume and ROCKET arguments needed to manage its own HTTPS encryption. Getting my test install of Caddy going was more tricky. I tried to explicitly disable its web server, but figured it was too much trouble for a mere test, so I moved on to working with containers.

While trying to spin up a Caddy container alongside Pi-Hole, I ran into something called rootlessport hogging port 8000. I ran updates and rebooted the server. And then I realized I was trying to put both Caddy and Vaultwarden on the same port! I got the two running at the same time and arrived on Caddy’s slanted welcome page both with IP and via Pi-Hole-served domain_name:port_number.

Subdomains are my next target. I mounted a simple Caddyfile pointing to Vaultwarden and got stuck for a while researching how I was going to forward ports 80 and 443 to 8000 and 44300, respectively. Long story short, I examined an old command I used to forward DNS traffic to Pi-Hole and after a much background research about other communication protocols, I decided to forward just TCP and UDP. I left myself a note in my administration home directory.

DNS: Domain Name System – Finds IP address for URL’s.

sudo firewall-cmd –zone=public –add-forward-port=port=8000:proto=tcp:toport=8000 –permanent
sudo firewall-cmd –zone=public –add-forward-port=port=8000:proto=udp:toport=8000 –permanent
sudo firewall-cmd –zone=public –add-forward-port=port=44300:proto=tcp:toport=44300 –permanent
sudo firewall-cmd –zone=public –add-forward-port=port=44300:proto=udp:toport=44300 –permanent

I still don’t get a reply from vaultwarden.buttonmash.lan. I tried nslookup, my new favorite tool for diagnosing DNS, but from observing Caddy’s cluttered logs, I spotted it rejecting my domain name because it couldn’t authenticate it publically. I found a “directive” to add to my declaration of reverse proxy to use internal encryption.

But I still couldn’t reach anything of interest – because reverse-proxied traffic was just bouncing around inside the Caddy container! The easy solution –I think– would be to stack everything into the same pod. I still want to try keeping everything in separate containers though. Another easy solution would be to set the network mode to “host,” which comes with security concerns, but would work in-line with what I expected starting out. However, Podman comes with its own virtual network I can hook into instead of lobbing everything onto the host’s localhost as I have been doing. Learning this network will be my goal for tonight’s session.

Tuesday Night

The basic idea behind using a Podman network is to let your containers and pods communicate. While containers in a pod communicate as if over localhost, containers and pods using a Podman network communicate as if on a Local Area Network down to ip address ranges.

My big question was if this was across users, but I couldn’t find anyone saying one way or the other. Eventually, I worked out a control test. Adding the default Podman network, “podman,” to the relevant start scripts, I used ip a where available to find containers’ ip addresses.Pi-Hole then used curl to grab a “Hello World!” hosted by Caddy on the same user. I then curled the same ip:port from Vaultwarden’s container and failed to connect. This locked-down behavior is expected from a security point of view.

On this slight downer, I’m going to call it a night. My goal for tomorrow is to explore additional options and settle on one even if I don’t start until the day after. In the rough order of easy to difficult (and loosely the inverse of my favorites), I have:

  1. Run Caddy without a container.
  2. Run Caddy’s container rootfully.
  3. Run Caddy’s container in network mode host.
  4. Move all containers into a single user.
  5. Perform more firewalld magic. (Possibly a flawed concept)
  6. (Daydreaming!!) Root creates a network all users can communicate across.

Whatever I do, I’ll have to weigh factors like security and the difficulty of maintenance. I want to minimize the need for using root, but I also want to keep the separate accounts for separate services in case someone breaks out of a container. At the same time, I need to ask if making these connections will negate any benefit for separating them across accounts to begin with. I don’t know.

Wednesday Afternoon

I spent the whole thing composing a help request.

Wednesday Night

The names I am after for higher-power networking of Podman containers are Netavark and Aardvark. Between 2018 and around February 2022 it would have been Slirp4netns and its plethora of plugins. Here approaching the end of 2023, that leaves a8 and onword is an outright betrayal round four years worth of obsolete tutorials to not quite two years with the current information – and that’s assuming everyone switched the moment the new standard was released, which is an optimistic assumption to say the least. In either case, I should be zeroing in on my search.

Most discouraging are how most of my search results involving Netavark and Aardvark end up pointing back to the Red Hat article announcing their introduction for fresh installs in Podman 4.0.

My goal for tomorrow is to make contact with someone who can point me in the right direction. Other than that, I’m considering moving all my containers to Nextcloud’s account or creating a new one for everything to share. It’s been a while since I’ve been this desperate for an answer. I’d even settle for a “Sorry, but it doesn’t work that way!”

Thursday Morning

Overnight I got a “This is not possible, podman is designed to fully isolate users from each that includes networking,” on Podman’s GitHub from Lupa99, one of the project maintainers [1].

Thursday Afternoon

Per Tuesday Night’s entry, I have multiple known solutions to my problem. While I’d love an extended discourse about which option would be optimal from a security standpoint in a production environment, I need to remember I am running a homelab. No one will be losing millions of dollars over a few days of downtime. It is time to stop the intensive researching and start doing.

I settled on consolidating my containers into one user. The logical choice was Pi-Hole: the home directory was relatively clean, I’d only need to migrate Vaultwarden. I created base directories for each service noting how I will need to make my own containers some day for things like games servers. For now, Pi-Hole, Caddy, and Vaultwarden are my goals.

Just before supper, I migrated my existing Pi-Hole from hard-mounted directories to Podman volumes using Pi-Hole’s Settings>Teleporter>Backup feature.

Thursday Night

My tinkerings with Pi-Hole were not unnoticed. At family worship I had a couple family members reporting some ads slipping through. At the moment, I’m stumped. If need be, I can remigrate by copying the old instance with a temporary container and both places mounted. My working assumption though is that it’s normal cat and mouse shenanigans with blocklists just needing to update.

It’s been about an hour, and I just learned that any-subdomain.buttonmash.lan and buttonmash.lan are two very different things. Every subdomain I plan to use on ButtonMash needs to be specified on PiHole as well as Caddy. With subtest.buttonmash.lan pointed at Caddy and the same subdomain pointed at my port 2019 Hello World!, I get a new error message. It looks like port 80 might be having some trouble getting to Caddy…

$ sudo firewall-cmd –list-all
…
forward-ports:
port=53:proto=udp:toport=5300:toaddr=
…

That would be only Pi-Hole’s port forward. Looking at that note I left myself Tuesday, and I can see I forwarded ports 8000 and 44300 into themselves! The error even ended up in the section above. Here’s the revised version:

sudo firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toport=8000 --permanent
sudo firewall-cmd --zone=public --add-forward-port=port=80:proto=udp:toport=8000 --permanent
sudo firewall-cmd --zone=public --add-forward-port=port=443:proto=tcp:toport=44300 --permanent
sudo firewall-cmd --zone=public --add-forward-port=port=443:proto=udp:toport=44300 --permanent

I also removed Tuesday’s flubs, but none of these changes showed up until I used

sudo firewall-cmd --reload

And so, with Pi-Hole forwarding subdomains individually and the firewall actually forwarding the HTTP and HTTPS ports (never mind that incoming UDP is still blocked for now), I went to https://vaultwarden.buttonmash.lan and was greeted with Firefox screaming at me, “Warning: Potential Security Risk Ahead” as expected. I’ll call that a good stopping point for the day.

My goal for tomorrow is to finish configuring my subdomains and extract the keys my devices need to trust Caddy’s root authority. It would also be good to either diagnose my Pi-Hole migration or re-migrate it a bit more aggressively.

Friday Afternoon

To go any farther on, I need to extract Caddy’s root Certificate Authority (CA) certificate and install it into the trust store of each device I expect to access the services I’m setting up. I’m shaky on my confidence here, but there are two layers of certificates: root and intermediate. The root key is kept secret, and is used to generate intermediate certificates. Intermediate keys are issued to websites to be used for encryption when communicating with clients. Clients can then use the root certificate to verify that a site’s intermediate certificate is made from an intermediate key generated from the CA’s root key. Please no one quote me on this – it’s only a good-faith effort to understand a very convoluted ritual our computers play to know who to trust.

For containerized Caddy installations, this file can be found at:

/data/caddy/pki/authorities/local/root.crt

This leads me to the trust command. Out of curiosity, I ran trust list on my workstation and lost count around 95, but I estimate between 120 and 150. To tell Linux to trust my CA, I entered:

trust anchor <path-to-.crt-file>

And then Firefox gave me a new warning: “The page isn’t redirecting properly,” suggesting an issue with cookies. I just had to correct some mismatched ip addresses. Now, after a couple years of working toward this goal, I finally have that HTTPS padlock. I’m going to call it a day for Sabbath.

My goal for Saturday night and/or Sunday is to clean things up a bit:

  • Establish trust on the rest of the home’s devices.
  • Finish Vaultwarden migration
  • Reverse Proxy my webUI’s to go through Caddy: GoldenOakLibry, PiHole, Cockpit (both ButtonMash and RedLaptop)
  • Configure Caddy so I can access its admin page as needed.
  • Remove -p ####:## bound ports from containers and make them go through Caddy. (NOT COCKPIT UNTIL AVAILABLE FROM REDUNDANT SERVER!!!)
  • Close up unneeded holes in the firewall.
  • Remove unneeded files I generated along the way.
  • Configure GoldenOakLibry to only accept connections through Caddy. Ideally, it would only accept proxied connections from ButtonMash or RedLaptop.
  • Turn my containers into systemd services and leave notes on how to update those services
  • Set up a mirrored Pi-Hole and Caddy on RedLaptop

Saturday Night

Wow. What was I thinking? I could spend a month in and of itself chewing on that list, and I don’t see myself as having the focus to follow through with everything. As it was, it took me a good half hour to just come up with the list.

Sunday

I didn’t get nearly as much done as I envisioned over the weekend because of a mental crash.

Nevertheless, I did do a little additional research. Where EndeavourOS was immediately recipient to the root certificate such that Firefox displayed an HTTPS padlock, the process remains incomplete from where I tried it on PopOS today. I followed the straightforward instructions found for Debian family systems on Arch Wiki, but when I tell it to update-ca-certificates, it claims to have added something no matter how many times I repeat the command without any of the numbers actually changing. I’ve reached out for help.

Monday Morning

I’ve verified that my certificate shows up in /etc/ssl/certs/ca-certificates.crt. This appears to be an issue with Firefox and KDE’s default browser on Debian-based systems. I’ll decide another week if I want to install the certificate directly to Firefox or if I want to explore the Firefox-Debian thing further.

Takeaway

Thinking back on this week, I am again reminded of the importance of leaving notes about how to maintain your system. Even the fog:head AM brain is better able to jot down a relevant URL that made everything clear where the same page may be difficult to re-locate in half a year.

My goal for next week is to develop Nextcloud further, though I’ll keep in mind the other list items from Friday.

Final Question

What do you think of my order of my list from Friday? Did I miss something obvious? Am I making it needlessly overcomplicated?

Let me know in the comments below or on my Socials!

Works Cited

[1] Shadow_8472, Luap99, “How Do I Network Rootless Containers Between Users? #20408,” github.com, Oct. 19, 2023. [Online]. https://github.com/containers/podman/discussions/20408. [Accessed Oct 23, 2023].

[2]. Arch Wiki, “User:Grawity/Adding a trusted CA certificate,” archlinux.org, Oct. 6 2022 (last edited), [Online]. https://wiki.archlinux.org/title/User:Grawity/Adding_a_trusted_CA_certificate#System-wide_–_Debian,_Ubuntu_(update-ca-certificates). [Accessed Oct 23, 2023].

Posted on

Never Underestimate Your Gremlins

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am working on my home network. Let’s get started!

Where to begin? Last week I left off with Puppy Linux. Well, I successfully installed it to a USB. While hardening FireFox, I noticed that the popular search-engine/online-advertising company is pushing out a new set of standards for their popular browser called Manifest 3 that will cripple functionality browser-based ad blockers rely on to keep prying eyes out (all in the name of privacy, of course); Mozilla/FireFox will be adopting these standards, with roll out this month: January 2023.

Network Collapse

In response, I prioritized setting up PiHole, a network-based ad blocker which won’t be affected by Manifest 3 and will work on Android devices. I soon learn it’s available in an OCI/“Docker” container. Long story short, I install it to ButtonMash and my old laptop for logistical reasons involving my dormant Family Photo Trunk project. I went to adjust the router’s DNS (Domain Name Server) settings to point at my PiHole containers figuring the worst that could happenwould be I just need five minutes tops to revert changes… the router moved itself from 192.168.0.1 to 162.168.1.1, collapsing the home network – including the workstation I was planning on using to fix it!

I was more than a bit stunned. Lucky for me, my old laptop was on a static IP address; unlucky: Bitwarden password manager has been a pain on that machine as of late, so I had to copy it manually from elsewhere. Once I was in, I reverted the DNS settings to automatic and most computers recovered by toggling network off and on (or rebooting) to refresh the automatic DHCP settings.

Upstairs Workstation

A while back, I rigged up a Raspberry Pi to work as a Wi-Fi catcher/subnet router, and it’s served me well up to this point. I switched its static, subnet-facing IP so it didn’t conflict with the one now claimed by the router, but as Iwas researching how to adjust its DHCP settings for the new subnet, I noticed its base operating system is at least months past end-of-life.  

Takeaway

I need to stop quoting optimistic worst-case scenarios. Gremlins can and will make a fool of me. On the other hand, I’m very thankful I had my laptop-server still able to navigate the crippled network with its static IP.  

I’ll be keeping the router where it is and see how saving the band of 192.168.0.* for static IP’s plays out. I guess I have the rest of this month’s projects planned out…

Final Question

What is the biggest computer oops you’ve ever had (and recovered from)?

Furthermore

I had a small adventure getting this post from LibreOffice on my upstairs workstation over to my blog without Internet. The way my filesystem is set up, it the save feature hangs badly when a mounted network drive doesn’t respond. I ended up using a .txt file on a thumb drive, and dumping it to the command line with cat, a terminal program to concatenate.

Privacy Policy