Category: Router

Posted on

Responsibility of the Network

Good Morning from my Robotics Lab! This is Shadow_8472, and today I have a doozy of a network week to cover. Let’s get started!

Meet the Computers

  • Cerberus – the main star today. It is our new hardware firewall running OPNsense
  • Red Router – a tp-link gaming Wi-Fi router fancied up beyond what it should have been
  • LAB – my homelab with a few servers
  • LAN – everything else connected via Wi-Fi or Ethernet

Network Implosion

It all started with revisiting a .lan domain. Cerberus’ extensive webUI left me with the hunch I’d need one machine in charge of DHCP assigning dynamic IP addresses. Red Router’s “operation mode” to work as an access point was hidden in literally the last menu to click through.

It was afternoon and no one would be using the network for the 30 seconds to 5 minutes I estimated switching the LAB and LAN Ethernet cables from Red Router over to Cerberus would take. Nope. No traffic made it through. DHCP mis-configuration? Cue a slow back and forth, bopping a setting from a workstation and trying a different physical configuration. Eventually, Cerberus ended up on my desk with Red Router talking directly to our ISP’s gateway/modem.

Order of events is a bit fuzzy from here, but when the Wi-Fi stopped working, I was without a good access to online answers. I worked the problem into the night. Around 1:00 AM, I knew I had done too much for a clean reversion. For two hours, I worked in loops hoping to spot something different. So much waiting! Cerberus would behave on my desk, then fail when redeployed. Worse: when I re-connected all the wires to Red Router, it started dancing between 192.168.0.1 and 192.168.1.1 every 45 seconds or so. I configured its IP manually, but gave up on Internet by morning at 3:00 AM, preparing to concede to my father’s suggestion earlier about hiring a professional to untangle my mess.

The Next Day

Newtork Loop. In my brain fog, I had Red Router talking to itself on a cable leftover from removing Cerberus for the night. With the house to myself for several hours, I alternated between bursts of intense diagnostics and mental processing. Somewhere in there, I rebooted the ISP’s modem.

Around noon, I realized the extra ports on Cerberus aren’t a switch as is Red Router’s default configuration, but were following firewall rules – which explained its behavior the previous day when I tried a computer from LAB without anything in-between. At around 3 PM, I got a Discord notification while mentally checked out, letting me know the network was back on.

6 PM on the second day: I situated my workstation in Cerberus’ LAN port and a Raspberry Pi in one I named LAN2. I’d previously copied firewall rules from LAN to LAN2 and LAB, but to no obvious effect – until I had the two computers ping each other. LAN2 failed as expected, but LAN’s ping was returned. I corrected the interfaces’ rules to allow them to reach out, and that was it.

Fallout

Without going into too much detail, a subnet shift like this is a major undertaking for networks with static IP servers on them. Not only do the network and computers need to be adjusted, but all traces of the old subnet need to be corrected. NFS clients needed to be told where the server was now, and the NFS server shares needed to be updated about what IP’s were allowed to mount them. I also still have Bitwarden to clients to update at my leisure.

Takeaway

OPNsense is a heavy weight in terms of configuration options. It has a learning curve compared to products simple enough to for Grandma and Grandpa to use. I may have solved my own emergency, but it may be wise to get someone looking at it professionally anyway to grade my work and give me some pointers on rootless Podman mounting NFS shares, or other long-term places where I’ve gotten stuck.

Final Question

I admit: networking is more fun than I gave it credit for before I knew basically anything. I still find it a bit taxing to mentally reach around my mental map, but I manage. How do you visualize networks?

Posted on

Learning OPNsense: DNS Adblocking

Good Morning from my Robotics Lab! This is Shadow_8472 with another short update my progress with my OPNsense firewall. Let’s get started.

I have done a substantial amount of work with DNS on my home network, but as noted in a previous post, it’s sub-optimal to exclusively manipulate your Domain Name System access from a power-hungry desktop when you sometimes have to ration electricity in your Uninterruptible Power Supply (UPS). I like PiHole’s web interface with all its fancy, moving graphs and charts, but our new firewall, Cerberus, can replicate the functionality I need.

I primarily use PiHole for DNS ad blocking, but I also explicitly blacklist a few URL’s while hosting local DNS records for servers on my Local Area Network (LAN), though the later is a work in progress.

OPNsense→Services→Unbound DNS→Blocklist→Type of DNSBL offers a drop-down checkbox menu of block lists. This is in contrast to PiHole→Adlists, which lets you import lists from arbitrary sources (edit the day after posting: OPNsense Unbound does have a URLs of Blocklists field). Either way, it should go without saying that sites and ads only need to be blocked once; it will only slow your DNS service down if given a bunch of redundant lists. From what I remember installing OISD Big onto PiHole, they aggregate several of these lists and remove the duplicates. PiHole also picked up a list named StephenBlack with a comment, “Migrated from /etc/pihole/adlists.list.” It sounds like a system default, but in any case, I found it had stuff not on OISD Big. OPNsense Unbound has the option for it, so it got migrated.

Migrating singled-out blacklist items was as simple as adding each entry to a comma-separated list (where PiHole wants separate entries). I’m going to wait on migrating my LAN domain names though. I believe I found the place to do it, but ButtonMash isn’t running Caddy to recognize subdomain requests right now.

One last step was to get into the red gaming router we’ve been using and point its DNS at Cerberus the Firewall. I then pointed its secondary DNS alternative at ButonMash.

To summarize, we should have the exact same protection as before on a smaller battery footprint and within the firewall’s default attack surface to boot!

Encrypted DNS

One of my eventual goals is to have my own recursive DNS server, which seeks out an a URL’s authoritative DNS record if it doesn’t have it cached. This will increase privacy, but I haven’t figured it out at a production grade yet. Instead, I looked up the best free and privacy respecting DNS, and so far as I can tell, that’s Cloudflare at 1.1.1.1.

From OPNsense, it wasn’t much more trouble to encrypt using DNS over TLS. I would prefer DNS over HTTPS, does the same thing but camouflages DNS requests as normal web traffic. For now, I’m assuming Unbound can’t do this and working properly. Please tell me if I’m wrong.

Takeaway

It’s slow going, but I am moving into Cerberus. While looking around, I found a module for NUT (Network UPS Tools), a utility for shutting down computers gracefully as their UPS runs down. I wanted to get it working, and for a moment after a reboot I did, but for reasons beyond me besides the driver on BSD not agreeing the best with CyberPower UPS systems, I’m at a loss. At this point, I am thinking to install a small Linux box to do the job at a future date, even though that will be yet another thing on the UPS.

Final Question

From above: Do you know of a way for OPNsense’s Unbound module to run DNS over HTTPS? I look forward to hearing from you in the comments below or on my Socials!

Posted on

Hardware Firewall Up!

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

I left off last week having made attempts on four separate nights trying to get the hardware firewall online in a production context. When I tested it between my upstairs workstation and its OpenWRT+Raspberry Pi router/Wi-Fi adapter, it worked fine. Put it back in production between our ISP’s gateway and our existing gaming router, and no one gets Internet.

The solution: pull the gateway’s plug for 30 seconds and let it reboot. Internet solved.

Longer explanation: my ISP box is in some sort of bridge mode, where it’s supposed to pass the external IP address to a single device (usually a router, but can be a normal computer). In this mode, it didn’t like this device getting swapped out – possibly as a security measure. It still reserves the address 10.0.0.1 as itself through out the network, a behavior I took to be half-bridge mode, but my surprise this week while fiddling with settings was that it did in-fact pass on the external address.

Takeaway

I expected the struggle to continue a lot longer, but I actually figured it out pretty quickly once I started researching the symptoms online. I explored the settings a bit more. I’d like to move the functions of PiHole over, but the web interface has a drop-down menu for block lists instead of a text box. I’ll look into it another time. Instead, I spent a good chunk of the week weeding grass and getting a sunburn.

Final Question

Have you ever found you were rebooting the wrong thing? I look forward to hearing from you in the comments below or on my Socials!

Posted on

Call the Electrician

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

My house has been having intermittent electrical issues for a while, but they flared up enough to diagnose a couple weeks ago. It affected our computer room and the downstairs/backyard lights. Kitchen appliances and most other outlets were on other breaker circuits, thankfully.

It was Monday afternoon. Last week’s post was up, and I was tentatively researching February’s big project. My sister, Taz, asked for help moving her workstation to another room and off the faulty circuit. She warned me about her outlet sparking, so I equipped some leather gloves before I began jiggling out her UPS.

UPS:
Uninterruptible Power Supply.

The outlet crackled as blue arcs flashed within the empty, upper socket and along the blades of the UPS plug as I wrestled with it. The room lights flickered off a few times, adding to the gut feeling I was aboard a starship, working over a console ready to explode in my face.

I put an amount of effort befitting a temporary relocation while arranging Taz’s computer. The challenge was her Internet connection. With OpenWRT fresh in my mind, the hardest part was deciding on a previous project SD card to overwrite.

Each card’s identity was easily confirmed by mounting it and checking <disk>/etc/label. One Raspberry OS install stuck out as redundant now that I use BalenaEtcher for disk imaging.

We quickly ran into an important gap in my networking knowledge: DNS. I shelved PiHole last week in part because I couldn’t get OpenWRT’s DHCP server to properly advertise it as the DNS server. Manjaro/KDE was mostly in last week’s attempts – even allowing for a DHCP IP with a manual DNS, but Windows’ “manual” IP configuration got wiped each time I tested “automatic (DHCP)”. This led to confusion and frustration when Taz saw our hosted Minecraft server online, but not the authentication servers for lack of DNS.

DNS:
Domain Name Service – translates URL’s into IP addresses, a “phone book” for computers

DHCP:
Dynamic Host Configuration Protocol – Automatic IP address configuration, a matradee for networks.

The electrician gave our sparking outlet a checkup. He said that as first goodie on the breaker circuit, everything else is wired in series. When it shorts, everything else is affected before breaker pop. Our outlet has buckled under a heavy load over years’ time; we’ll minimize its usage until repairs happen this coming week. Fun fact: lights and sockets sharing a breaker is a code violation, but it could have been fine when the house was built.

In the meantime, I idled a day or two while trying/failing to fix my DHCP configuration for lack of search terms. OpenWRT’s Lu-Ci web interface strikes a good balance: it’s user-friendly without being baby or admin-proofed. Nevertheless, I took my issue to r/techsupport’s Discord, where I learned about DHCP-options. So far as I can tell, DHCP-options is just a lookup table. Option 6 specifies a list of IPv4 addresses as DNS servers:

6,192.168.0.2,192.168.0.20

Takeaway

Messing with a sparking outlet the way I did was stupid. A few days’ retrospect told me I should have had a fire extinguisher ready. Editing this the night before posting, it dawns on me that de-energizing the whole breaker circuit would have been better still. I’m thankful nothing happened and that the situation is stable enough to wait for a repair.

I’m also glad to have learned about DHCP options. Of note, I picked up this week that Raspberry Pi Wi-Fi radios were never going to win any performance prizes. My Internet slowdowns are not just me.

Final Question

I’m trying something new by isolating glossary terms in a column. They were a pain to figure out, but I think I can control them now. What do you think?

Let me know in either the comments below or on my Socials.

Posted on

Networking Is Magic

Good Morning from my Robotics Lab! This is Shadow_8472, and if you missed last week’s post, feel free to check it out. It… was a minor disaster. I tried setting up PiHole network adblocker, but my home router unexpectedly moved its local IP address in the process. I cleaned up what I could really quickly and noticed my little hackjob of a subnet router was running an end-of-life operating system. Today, I am fixing that oversight. Let’s get started!

Replacing My Hackjob Router

Hackjob Router was my consolation prize after a failed quest to implement the open source router firmware, OpenWRT, onto my Raspberry Pi 4B in early 2020 and another in 2021. Both times, the exact version I needed was still under development. Dealing with testing versions was too advanced of a magic spell for me. I did, however, find an easy tutorial within my reach, but it did little to advance me beyond an aspiring networking mage with smoke and mirrors, but no fire or glass. When I looked this week, the beta warning was lifted.

I downloaded OpenWRT and flashed it over Hackjob Router’s SD card. Sure enough, the web interface was complete. I’ve used at least half a dozen ranging from limited config options to full access. OpenWRT’s “Lu-Ci” web interface puts everything on display with a helpful tool tip. It is comparable to other network devices I’ve worked with, but is simpler to look at, and has slightly more functionality.

My final configuration was surprisingly easy for a project that’s been hanging for almost three years now. At no point did I gain a key insight directly from an online search. But mistakes were made, and background information was researched and shelved for later.

My first mistake was a wrong Wi-Fi password. When I finally located and corrected it, my connection to OpenWRT died. I quickly learned how to assign a static IP in KDE’s settings thanks to intuitive interface design. I researched br-lan, a virtual network interface used for assigning one IP to multiple physical interfaces, thinking I needed to add the physical Wi-Fi radio to the one automatically generated to host all of the one Ethernet port and “bridge” the two sides that way.

The problem was actually a bad netmask. IPv4 network addresses come in four eight-bit numbers between 0 and 255. Local networks mask off leading bits – typically in multiples of 8 (for example: 10.0.0.1/8). Routers can use DHCP to dynamically assign local IP addresses with in their assigned subnet. My subnet ranges between 192.168.1.0 and 192.168.1.255 – properly denoted 192.168.1.1/24. Originally, my trailing mask was /16, allowing DHCP to assign my workstation to 192.168.0.200. Correcting the mask made it behave.

An unanchored memory I have regarding this week’s research is that some devices can route packets directly between network interfaces as opposed to routing them manually. I doubt the Raspberry Pi 4 has this ability, but it would be nice to know for sure.

Takeaway

Networking is magic at times. I still have a long way to go before I understand enough to do everything I want to, but I’ve cleared a large and long-standing burrier toward that goal this week. This is in part thanks to OpenWRT’s Lu-Ci with its educational help tips about every drop-down menu, text field, and tick box.

Final Question

Do you ever study a known science and everything inside you insists it’s magic?

I look forward hearing your answers on in the comments below or on my Socials.

Posted on

I Switched My Operations to Caddy Web Server

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am rebuilding my home server, Button Mash, from the operating system up. Let’s get started!

Caddy Over Nginx

I spent well over a month obsessing over Nginx, so why would I start over now? As far as I am concerned, Caddy is the piece of software for my use case. While I am sure Nginx is great at what it does, I keep slamming into its learning curve – especially with integrating Let’s Encrypt, a tool for automating SSL encryption (HTTPS/the green padlock). Caddy builds that functionality in while still doing everything I wanted Nginx for.

The official Caddy install instructions [1] for Fedora, Red Hat, and CentOS systems are as follows:

$ dnf install ‘dnf-command(copr)’
$ dnf copr enable @caddy/caddy
$ dnf install caddy

First of all, new command: copr. Background research time! COPR (Cool Other Package Repositories) is a Fedora project I feel comfortable comparing to the Arch User Repository (AUR) or Personal Package Archive (PPA): it lets users make their own software repositories.

Installation went smoothly. When I enabled the repository, I had to accept a GPG key that wasn’t mentioned in the instructions at all. From a user point of view, they appear to fill a similar purpose here to a SSH keys: special numbers use math to prove you are still you in case you get lost.

Caddy uses an HTML interface (a REST API – Don’t ask, I don’t understand myself) on the computer’s internal network known as loopback or localhost on port 2019. Caddy additionally serves everything over HTTPS by default. If it cannot convince Let’s Encrypt to give it a security certificate, it will sign one itself and tell the operating system to trust it. In other words, if I were not running ButtonMash headless (without a graphical interface), I’d be able to try connecting to localhost:2019 with a favorite browser, like at least one of the limited supply of Caddy tutorials did.

IP Range Transplant

I should have just done my experimentation on DerpyChips or something. Instead, I pressed on with trying to point a family-owned domain name at Button Mash. This side adventure sprouted into last week’s post. In short: ButtonMash’s static IP kept was in conflict with what my ISP-provided equipment kept trying to assign it, resulting in an estimated 50% downtime from a confused router. Upgrading to the next gateway may have allowed us to free up the IP range for the gaming router’s use, but it’s not out for our area yet. My father and I switched our network connections over to a “gaming router” we had laying about and enabled bridge mode on the gateway to supposedly disable its router part. I have my doubts about how it’s actually implemented.

Most of our computers gladly accepted the new IP range, but GoldenOakLibry and ButtonMash –having static IP’s– were holdouts. I temporarily reactivated a few lines of configuration on my laptop to set a static IP so I could talk with them directly and manually transfer them over to the new IP range, breaking NFS shares and Vaultwarden on them respectively.

In the confusion, ButtonMash lost its DNS settings; those were easy enough to fix by copying a config line to point those requests to the router. GoldenOakLibry took a bit longer to figure out because the NFS shares themselves had to accept traffic from the new IP range with settings buried deep within the web interface. Once that was sorted, I had to adjust the .mount files in or around /etc/systemd/system on several computers. Editing note: While trying to upload, I found I could not access GoldenOakLibry on at least a couple of my machines. Note 2: I had to change the DHCP settings to the new IP range on my Raspberry Pi reverse Wi-Fi router. Systemd on both goofed systems needed a “swift kick” to fix them.

sudo systemctl start <mount-path-with-hyphens>.mount

Repairs Incomplete

That left Vaultwarden. I was already in a it’s-broken:-fix-it-properly mentality from the modem/router spinoff project. I got as far as briefly forwarding the needed ports for an incompletely configured Caddy to respond with an error message before deciding I wanted to ensure Bitwarden was locked down tightly before exposing it to the Internet. That wasn’t happening without learning Caddy’s reverse proxy, as I put Vaultwarden exclusively onto a loopback port.

Speaking of loopback, I found the official Caddy tutorials lacking. They –like many others after them– never consider a pupil with a headless server. I have not yet figured out how to properly convince my other computers to trust Caddy’s self-signed certificates and open up the administration endpoint. That will come in another post. I did get it to serve stuff over HTTP by listing IP’s as http://<LAN address>, but Bitwarden/Vaultwarden won’t let me log in on plain HTTP, even over a trusted network and confine the annoying log to a file.

As far as I can tell, the administration API on port 2019 does not serve a normal web page. Despite my efforts, the most access I have gotten to it was for it to error out with “host not allowed.” I haven’t made total sense of it yet. I recognize some of the jargon being used, but its exact mechanics are beyond me for the time being.

Takeaway

Caddy is a powerful tool. The documentation is aesthetically presented and easy enough to understand if you aren’t skimming. But you will have a much better time teaching yourself when you aren’t trying to learn it over the network like I did.

Final Question

Do you know Caddy? I can tell I’m close, but I can’t know for sure if I’m there in terms of the HTTP API and just don’t recognize it yet. I look forward hearing from you in the comments below or on my Discord server.

Works Cited

[1] “Install,” Caddy Documentation. [Online], Available: https://caddyserver.com/docs/install. [Accessed: June 6, 2022].

Posted on

Patience is a Virtue When Internet Breaks

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

It’s the last Monday of the month. I should have a larger project this week, but I had to split a step off into a prequel of sorts when it ballooned on me. For reasons I will go into next week, I ended up talking with tech support for my Internet provider.

It all started when I logged into my router’s off-site network settings (WHY??? is this a thing?) so I could forward a couple ports to Button Mash, my home server, when I noticed it alternating between two internal IP’s: the static IP I configured it to call itself and another one I thought was assigned to another [possibly decommissioned at present] Raspberry Pi – or it may have been a setting I never fully cleaned up. I have no way to tell.

Additionally, I learned that the online toy they’re having me use to try and manage the network is being retired. They want me to use their app I don’t want on the phone I don’t carry. No thanks. I’d rather move my whole network over to this 3rd party gaming router we have on hand.

Support Begins

Friday before posting: On my first session with support, I got a live agent after complaining to their help bot that it hated me. When I shared how I wanted to use my own router, the agent mentioned “bridge mode,” an option I had noted in the gateway’s admin panel. We continued by poking at my Manjaro workstation’s 10 Mbps connection speed with little luck because I was using a different computer.

Sunday: I got into chat with an agent on my now 3.5 Mbps connection. To not-my surprise, the obligatory reboot everything between me and the Internet didn’t change much. I told him no fewer than two or three times that I was on desktop, not phone; I do not have mobile data to remain connected; no, I do not think my family is interested in a coupon for this streaming service you’re offering; this is my one and only means of connecting to you. I followed my given instructions and pressed a button on the back of the gateway for an unusually long time. The Internet did not come back on.

Level 2

My father called a number given to me by the second chat agent, and we were connected with an awesome “Level 2” agent who could follow along as I described my unique home network while repairing the damage done by a factory reset. When bridge mode came up, I figured we might as well fix it how we want if it’s already broken and we have a professional on the line. He had some sudden technical issues of his own an hour plus into helping us.

We called back and got someone whom I had to remind a couple times that Debian 10 does not mean Windows 10. Nevertheless, he got the idea through to us that moving/disabling the default IP range was not a feature our gateway supports. I read something as much in a forum post; it said something about Layer 2 network devices, but that’s homework for a future topic.

The first guy called back just as I was giving into despair with the second guy. Long story short: our network is at around 80%; ButtonMash and GoldenOakLibry, our server and network storage, are configured with static IP’s and don’t show up at all right now. The awesome agent suggested upgrading to the next router, which might not be hard coded to serve its default IP range, allowing me to swap IP ranges between gateway and router. I don’t want to manually configure all my computers to find GoldenOakLibry at a new IP.

Takeaway

A lot of going through tech support as a customer is about explaining your exact situation to however many agents you may come into contact with.

Special thanks to Mr. E on my family’s personal Discord server for the suggestion of disabling the gateway’s integrated router.

Final Question

Have you ever had tech support struggling to keep up with you?

I look forward hearing your answers on in the comments below or on my Discord server.

Posted on

Family Photo Chest Part 13: Early Prototype Workflow

Good Morning from my Robotics Lab! This is Shadow_8472, and today, I am merging a couple projects into one and hoping they stick. Let’s get started!

Overview

I’ve been piecing bits of my photo trunk project workflow together now for way too long. Right now, the architecture is looking like I’ll be scanning sets of pictures to a directory on a Network Attached Storage, then I can use a cluster of dedicated microcomputers from another unfinished project to separate and deskew the raw data into individual files. These files will then live permanently on the NAS.

Progress is rarely linear though. My end goal for this week hopped around quite a bit, but in the end I felt like I did nothing but figuring out how not to proceed: ground work with no structure. Routers are hard when you’re trying to learn them on a schedule!

Lack of Progress

In a perfect world, I would have been well on the way to configuring a cluster node by now. In a less unreasonable one, I would have had my Pi 4 OpenWRT ready to support the cluster. Late-cycle diagnostics chased me into an even more fundamental problem with the system: Wi-Fi connectivity.

During diagnostics, I’m learning about how different parts of the system work. Physical connection points can be bridged for a single logical interface, and Ethernet cables can support separate ipv4 and ipv6 connections. I can’t configure the Ethernet (on either logical interface) the way I want because that’s how I’m connecting to the Web UI and SSH. I end up stuck using two computers besides the two router Pi’s (OpenWRT and a Raspian hack router that actually works) because I don’t like switching my Ethernet cables around on the switch, but I need to do that anyway when I have to copy a large block of text. In short: the sooner Wi-Fi gets working, the better.

I understand I am essentially working with a snapshot. It’s been tidied up a bit, but bugs still exist. Wi-Fi is apparently one of those things that’s extra delicate; each country has its own region, among other complexities. On the other hand, I don’t know if that’s actually the case, as diagnostics are ongoing.

Takeaway

I’m probably going to work on this one in the background for a while. The OpenWRT help forum’s polish at least in part makes up for routers being dull to learn. If it takes too long, I do have other projects, so I may need to replace the cluster with a more readily available solution.

Final Question

Have you ever had upstream bugs that kept you from completing a task?

Privacy Policy