Category: Networking

Posted on

Podman-Nextcloud: Climb Shorter Mountains

Good Morning from my Robotics Lab! This is Shadow_8472 and today it’s bully or be bullied as I take another swing at getting my Nextcloud instance even partially usable. Let’s get started!

If there’s one long term project of mine that just loves humiliating me, it’s getting Nextcloud operational. My eventual goal is to have it running in a rootless Podman container with a way to quickly move it to an auxiliary server. My strategy thus far has been to prepare three Podman volumes hosted on GoldenOakLibry (NAS) over NFS while accounting for the speed needs of the MariaDB and Nextcloud volumes with an SSD and vs the capacity needs of the PhotoTrunk volume with a RAID 5 array of HDD’s.

NAS: Network Attached Storage

NFS: Network File System

SSD: Solid State Drive

HDD: Hard Disk Drive

Lowering Expectations

I’ve lost count of how many times NFS has given me grief on this project, so I eliminated it. I moved the SSD from where it was on GoldenOakLibry to ButtonMash, my main server computer. I added it to /etc/fstab – bricking Rocky Linux. ButtonMash is dual booted, so I booted to Debian for repairs.

Rocky’s installation system uses an LVM2 format, which Debian can’t read by default. An LVM2 package exists, and I installed it. LVM2 partitions show up in lsblk as sub-partitions of an actual partition, and it is these sub-partitions that get mounted, for example:

sudo mount /dev/rl_buttonmash/root /mnt/temp

to mount the sub-partition that shows up as rl_buttonmash-root. While I did explore for a quick fix, it’s a very good thing when each side of a dual booted machine can repair the other. Mounting a file system is a very important tool in that kit.

Upon closer inspection, a contributing factor to bricking Rocky was the root account being locked. The computer booted into an emergency mode and got stuck in a loop ending with “Press ENTER to continue…” Unlocking it didn’t get me anywhere when I looked at the logs per the loop’s recommendation, but the command lsblk -f clued me in that I was mounting the drive using the wrong file system type, an error which was soon remedied after I discovered it.

Project Impossible

The move hardly seemed to fix anything as hoped. I didn’t solve much. I kept getting NFS related errors when trying to run the pod, even after moving to a new mountpoint I’d never touched with NFS automounts. I even tried mounting the volumes using hard links pointed at the mounted “data drive” and I still couldn’t get a working Nextcloud instance. Somewhere in my shambling among the apparently limited content available regarding this topic online, I found the following warning on Oracle’s Podman documentation:

Caution: When containers are run by users without root permissions, Podman lacks the necessary permissions to access network shares and mounted volumes. If you intend to run containers as a standard user, only configure directory locations on local file systems [1].

Rootless Podman lacks network share permissions. OK, so NFS out unless I can selectively give Podman network permission without going full root. Until then, Podman is limited to the local disk, and if I’m understanding this warning correctly, mounted drives are also off the table. My plans for a Photo Trunk upgrade may be grounded indefinitely, and with ButtonMash’s Rocky drive being only 60GB, I’m not looking to burden it with anything resembling bulk storage.

Takeaway

The next logical innovation would be to rebuild the project on a computer with more storage. Barring a full makeover of ButtonMash, I do have my Red Laptop as an auxiliary server. I made a new account, but in all reality, this inspiration came after my research cutoff. It’s a project for another week once again.

Final Question

My project directory is messy with scripts to the point where I started a README file. Have you ever had a project so involved as to need one?

I look forward to hearing about it in the comments below or on my Socials.

Work Cited

[1] Oracle, “Configuring Storage for Podman,” oracle.com, [Online]. Available: https://docs.oracle.com/en/operating-systems/oracle-linux/podman/podman-ConfiguringStorageforPodman.html. [Accessed July 27, 2023].

Posted on

Call the Electrician

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

My house has been having intermittent electrical issues for a while, but they flared up enough to diagnose a couple weeks ago. It affected our computer room and the downstairs/backyard lights. Kitchen appliances and most other outlets were on other breaker circuits, thankfully.

It was Monday afternoon. Last week’s post was up, and I was tentatively researching February’s big project. My sister, Taz, asked for help moving her workstation to another room and off the faulty circuit. She warned me about her outlet sparking, so I equipped some leather gloves before I began jiggling out her UPS.

UPS:
Uninterruptible Power Supply.

The outlet crackled as blue arcs flashed within the empty, upper socket and along the blades of the UPS plug as I wrestled with it. The room lights flickered off a few times, adding to the gut feeling I was aboard a starship, working over a console ready to explode in my face.

I put an amount of effort befitting a temporary relocation while arranging Taz’s computer. The challenge was her Internet connection. With OpenWRT fresh in my mind, the hardest part was deciding on a previous project SD card to overwrite.

Each card’s identity was easily confirmed by mounting it and checking <disk>/etc/label. One Raspberry OS install stuck out as redundant now that I use BalenaEtcher for disk imaging.

We quickly ran into an important gap in my networking knowledge: DNS. I shelved PiHole last week in part because I couldn’t get OpenWRT’s DHCP server to properly advertise it as the DNS server. Manjaro/KDE was mostly in last week’s attempts – even allowing for a DHCP IP with a manual DNS, but Windows’ “manual” IP configuration got wiped each time I tested “automatic (DHCP)”. This led to confusion and frustration when Taz saw our hosted Minecraft server online, but not the authentication servers for lack of DNS.

DNS:
Domain Name Service – translates URL’s into IP addresses, a “phone book” for computers

DHCP:
Dynamic Host Configuration Protocol – Automatic IP address configuration, a matradee for networks.

The electrician gave our sparking outlet a checkup. He said that as first goodie on the breaker circuit, everything else is wired in series. When it shorts, everything else is affected before breaker pop. Our outlet has buckled under a heavy load over years’ time; we’ll minimize its usage until repairs happen this coming week. Fun fact: lights and sockets sharing a breaker is a code violation, but it could have been fine when the house was built.

In the meantime, I idled a day or two while trying/failing to fix my DHCP configuration for lack of search terms. OpenWRT’s Lu-Ci web interface strikes a good balance: it’s user-friendly without being baby or admin-proofed. Nevertheless, I took my issue to r/techsupport’s Discord, where I learned about DHCP-options. So far as I can tell, DHCP-options is just a lookup table. Option 6 specifies a list of IPv4 addresses as DNS servers:

6,192.168.0.2,192.168.0.20

Takeaway

Messing with a sparking outlet the way I did was stupid. A few days’ retrospect told me I should have had a fire extinguisher ready. Editing this the night before posting, it dawns on me that de-energizing the whole breaker circuit would have been better still. I’m thankful nothing happened and that the situation is stable enough to wait for a repair.

I’m also glad to have learned about DHCP options. Of note, I picked up this week that Raspberry Pi Wi-Fi radios were never going to win any performance prizes. My Internet slowdowns are not just me.

Final Question

I’m trying something new by isolating glossary terms in a column. They were a pain to figure out, but I think I can control them now. What do you think?

Let me know in either the comments below or on my Socials.

Posted on

My PiHole is “Half Baked”

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am installing PiHole. With luck, I’ll have be configuring some of its other functions to augment my home’s network as well. Let’s get started!

PiHole, Take II

I can rant about the evils of Google ‘til boredom do its part. However, this search engine is between inconvenient and impossible to ignore, given its impressive list of “hobbies” from STEM projects to smartphones. It’s an open secret few care to think about that their empire is built off user exploitation. I installed ad blocker browser plugins over their aggression last presidential election cycle.

Earlier this month, I read about Manifest v3, the new browser-plugin interface library created by Google. Their precautions against spyware just so happen to cripple ad blockers, among other legitimate plugins. This walking conflict of interest is set to roll out January, 2023, and Firefox is going along with it.

When a browser loads a web page, it asks a DNS service to translate the page’s URL into an IP address. It then finds, loads, and renders the page at that IP. This may involve loading other pages –such as ads– as elements of the original page. Network ad blockers protect you by fudging bad URL’s addresses.

Objectives

My main goal this week is to kill ads across my home network. Follow-up objectives include advanced PiHole features and a private DNS for even better protection.

Night 1

My first attempt at PiHole was messy. I set up PiHole OCI/“Docker” containers across my two servers – ButtonMash and my old laptop. Like before, the main router skipped IP’s on me. I had it repaired within an hour thanks to my same laptop functioning as a workstation with a static IP. With the router upgrade to my upstairs workstation, I easily archived its settings and outfitted it with its own wider network static IP – complete with a netmask wide enough to chase down its rogue counterpart should it shift again (Did I have laptop’s static IP netmask configured incorrectly this whole time?!).

Surprise! The expanded subnet didn’t work because the rogue router had its own subnet mask I was outside of. The dance was too involved for a play by play, but I only really felt helpless while trying to avoid hiking around to different workstations to clean up after this failed networking spell. As I reassembled the router for normal operation, I reasoned out that my router’s firmware is hardwired not to consider a DNS coming from a LAN connection, like I’m trying to do.

Flashing open source firmware is out of the question. For one, I wouldn’t know how to fix it and don’t have a replacement. Two: apparently its chipset manufacturer isn’t a fan of open source – the help thread I spotted recommended contacting OP’s government representative if he wanted to do anything about it.

Night 2

I did a bit of research before dismantling the network again. DHCP settings include optional fields for DNS requests. This should let me direct computers straight to PiHole instead of relaying the request in a convoluted workaround involving a NAT table and possibly causing a network loop.

This means each router is now a separate task. The responsible thing to do now is ensure my subnet router can behave before working on the main one. It’s not long before I fry my DNS settings. Navigation around my local network remains unaffected, but I eventually resort to restoring my backup from yesterday, re-applying the static IP, and updating the backup.

My best bet from here is to finalize my PiHole install. My initial container creation was the absolute minimum: port 80 web interface, port 53/TCP+UDP. There’s a lengthy list of environment variables to browse.

A Few Days Later

Jackpot! My mind cleared enough before bed to skim PiHole Docker’s documentation on GitHub. It has a list of example deployments – including a shell script. I converted it for Podman, entered my environment variables, and –during debugging– axed the logic for relaying logs as it was causing problems and I can view them directly with Cockpit-Podman.

PiHole User

But where to land it? I’ll eventually integrate as I master Caddy. Leaving the container running as root lets it use the proper ports, but I know better. Thanks to discoveries I spun off into last week’s project, I can now make more underprivileged, Cockpit-enabled users than I will ever need by using loopback the address (127.0.0.1/8).

The run script was easy to copy over to my new PiHole user. I gave it the directories it wanted as mountable volumes and shifted ports around until I was happy. I took the time to tidy up my firewall, combining a couple related entries and reclosing the normal DNS port.

I remember having issues with Vaultwarden’s stability over the course of days to weeks. The problem was occasionally annoying as Bitwarden only requires its home server when modifying the password vault, but PiHole will be sorely missed the moment it goes down. The one place I found the solution was in the official Podman troubleshooting guide on their GitHub [1]:

loginctl allow-linger userName

I sadly could not verify this was my previous, solution to my Vaultwarden long-term issues, but it’s not entirely unfamiliar, and it’s my best-informed guess.

DNS Port Forwarding

With PiHole secured in its own, easily accessible account, I soon experienced how picky DNS requests are about using the privileged port 53. All my attempts at manually telling OpenWRT to use port 5300 failed. I expect the the story will be the same if I try with on my main router.

I found the solution where Woody from b-woody.com blogged about almost the exact same project last May [2]: port forward port 53 to port 5300. Paranoid about goofing my firewall over command line I ran my version of Woody’s commands past r/TechSupport’s Discord channel. Moderator Donjuanal confirmed my omission of a trailing “:toaddr=”, but questioned my blind use of tcp, explaining how DNS clients default to udp for speed.

sudo firewall-cmd --zone=public --add-forward-port=port=53:proto=udp:toport=5300 --permanent

Even with this measure in place, I had to access the web console and tick Settings>DNS>Interface settings>Potentially dangerous options>Permit all origins before my local requests made it through. This may need to be addressed later.

Takeaway

I am so glad to have PiHole installed, even if it doesn’t appear to be doing much more than the uBlock Origin Firefox plugin. I’m researching the next segment though, and I estimate another week or more worth of work before it is configured alongside a private DNS server. Worth noting is that Firefox is leaving in the features ad block requires, despite potential security concerns. This is as good enough stopping point.

Final Question

Do you use PiHole? I’d be happy to hear about your experience.

I look forward hearing your answers on in the comments below or on my Socials.

Works Cited

[1] eriksjolund, “Podman\ Troubleshooting\ A list of common issues and solutions for Podman,” github.com, Nov. 19, 2022. [Online]. Available: https://github.com/containers/podman/blob/main/troubleshooting.md [Accessed Jan. 30, 2023].

[2] Woody, “Run PiHole in a rootless Podman container,” b-woody.com, May 12, 2022.[Online]. Available: https://b-woody.com/posts/2022-05-12-pihole-on-a-rootless-podman-container/ [Accessed Jan. 30, 2023].

[3] Can You Block It, “CAN YOU BLOCK IT?\ AN SIMPLE AD BLOCK TESTER” canyoublokit.com, 2021. [Online]. Available: https://canyoublockit.com/ [Accessed Jan. 30, 2023].

Posted on

I Glitched Cockpit and Discovered Multi-user Login

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project for the week. Let’s get started!

My mother needed an extra browser, so I installed Firefox hardened it a little. I took the liberty of adding the Bitwarden plugin, encouraging her to make an account on my self-hosted instance. Remembering my failure so far to diagnose the “Network Error” blocking log in, I spared the time to learn how new Bitwarden clients are slightly incompatible with old Vaultwarden servers.

I easily could have updated Vaultwarden with maybe a note on the blog Discord. Instead, I felt like adding VaultwardenUsr@localhost to Cockpit with “Add new host.” This stunt worked at the cost of forwarding shadow8472@ButtonMash to VaultwardenUsr@ButtonMash when to logging in. Relogging didn’t help, and the hosts list saw VaultwardenUsr as the primary login – disallowing me from removing it, and as a remote login – blocking my attempts to add my real primary account back in with the same stunt.

While exploring this bug, I logged into my old laptop server and linked its Cockpit back into ButtonMash without getting forwarded to VaultwardenUsr. At this point, I submitted a bug report to Cockpit’s GitHub. I soon found the malformed host list at /etc/cockpit/machines.d/99-webui.json. I backed it up, purged the malformed entry, and updated GitHub with my workaround.

Out of curiosity, I added VaultwardenUsr@192.168.0.— as an alternate host. This sends packets for an extra detour, but it works as required. Only after all this did I update my Vaultwarden image from Docker Hub and deploy a new container from it using the same command as the last two successful times.

Note: While working on next week’s project, I logged into VaultwardenUsr@127.0.0.1 and other loopback IP’s with no problems. It’s just name@localhost that causes problems.

Takeaway

1 day for the win! My push for PiHole and supporting network projects has been intense lately, so it’s great to have a smaller project where I still learn while by doing something important.

Final Question

Have you ever misused a software feature successfully? What challenges did you face before getting it to work how you had in mind?

look forward hearing your answers on in the comments below or on my Socials.

Posted on

Networking Is Magic

Good Morning from my Robotics Lab! This is Shadow_8472, and if you missed last week’s post, feel free to check it out. It… was a minor disaster. I tried setting up PiHole network adblocker, but my home router unexpectedly moved its local IP address in the process. I cleaned up what I could really quickly and noticed my little hackjob of a subnet router was running an end-of-life operating system. Today, I am fixing that oversight. Let’s get started!

Replacing My Hackjob Router

Hackjob Router was my consolation prize after a failed quest to implement the open source router firmware, OpenWRT, onto my Raspberry Pi 4B in early 2020 and another in 2021. Both times, the exact version I needed was still under development. Dealing with testing versions was too advanced of a magic spell for me. I did, however, find an easy tutorial within my reach, but it did little to advance me beyond an aspiring networking mage with smoke and mirrors, but no fire or glass. When I looked this week, the beta warning was lifted.

I downloaded OpenWRT and flashed it over Hackjob Router’s SD card. Sure enough, the web interface was complete. I’ve used at least half a dozen ranging from limited config options to full access. OpenWRT’s “Lu-Ci” web interface puts everything on display with a helpful tool tip. It is comparable to other network devices I’ve worked with, but is simpler to look at, and has slightly more functionality.

My final configuration was surprisingly easy for a project that’s been hanging for almost three years now. At no point did I gain a key insight directly from an online search. But mistakes were made, and background information was researched and shelved for later.

My first mistake was a wrong Wi-Fi password. When I finally located and corrected it, my connection to OpenWRT died. I quickly learned how to assign a static IP in KDE’s settings thanks to intuitive interface design. I researched br-lan, a virtual network interface used for assigning one IP to multiple physical interfaces, thinking I needed to add the physical Wi-Fi radio to the one automatically generated to host all of the one Ethernet port and “bridge” the two sides that way.

The problem was actually a bad netmask. IPv4 network addresses come in four eight-bit numbers between 0 and 255. Local networks mask off leading bits – typically in multiples of 8 (for example: 10.0.0.1/8). Routers can use DHCP to dynamically assign local IP addresses with in their assigned subnet. My subnet ranges between 192.168.1.0 and 192.168.1.255 – properly denoted 192.168.1.1/24. Originally, my trailing mask was /16, allowing DHCP to assign my workstation to 192.168.0.200. Correcting the mask made it behave.

An unanchored memory I have regarding this week’s research is that some devices can route packets directly between network interfaces as opposed to routing them manually. I doubt the Raspberry Pi 4 has this ability, but it would be nice to know for sure.

Takeaway

Networking is magic at times. I still have a long way to go before I understand enough to do everything I want to, but I’ve cleared a large and long-standing burrier toward that goal this week. This is in part thanks to OpenWRT’s Lu-Ci with its educational help tips about every drop-down menu, text field, and tick box.

Final Question

Do you ever study a known science and everything inside you insists it’s magic?

I look forward hearing your answers on in the comments below or on my Socials.

Posted on

I Survived Self-Hosting a Wiki With Podman!

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am setting up not one, but two personal wikis on my home network. Let’s get started!

A Personal Wiki

Wikis are the reference material of choice for the casual researcher in this day and age. The content of the subject encyclopedia is turbocharged by the power of the hyperlink when compared to a volume/page reference that can take minutes to weeks to “load” depending on circumstantial accessibility. Community contributions allow for information to be updated in a timely manner, while built-in version control helps admins quickly repair sabotage.

This technology can easily be deployed to a closed-off environment for personal, group, or enterprise use. I know I could use one to organize my role play games, and my sister is after one to help with her writing. My goal for this month’s large project is to get both these wikis operational within our home network.

Wiki Planning

The first list of open source wiki software listed Wiki.js as supporting deployment on Docker. If my Rocky Linux 8 experience with ButtonMash has taught me anything, it’s that OCI containers are good for easy cleanup of botched installations, though challenges can arise when using Podman instead of Docker.

I spent a day studying Wiki.js off and on. My basic understanding is that you need three things for a wiki: the web server, the database, and the wiki software itself. I already understand a bit about the relationship between website and web server. Database vs. website is a similar relationship to website vs. browser. It is an independent process that serves data a website garnishes before presenting to a browser. It’s even possible for multiple websites to share a common database. While Wiki.js currently supports a few SQL (Structured Query Language) databases, PostgreSQL is the only database they will support in future versions.

Sparing a thought for my photo trunk project, I believe a wiki has potential for distribution once we learn more about how to use one. ButtonMash is configured to use the scanner though on its Debian install and not the Rocky 8 one, so I’ll need a different machine to host. GoldenOakLibry comes to mind as its primary function is to host and serve files.

My First Operable Prototype Wiki

GoldenOakLibry is a Network Attached Storage (NAS) by Synology running a custom version of Linux called Disk Station Manager. I found MediaWiki, the wiki software powering Wikipedia.org, in its package manager and chose the easy route. I did not know what I was doing for the most part – just that I was glad I re-enabled the old Vaultwarden container to store new passwords I made as I passed through Password Purgatory: database, wiki root, wiki admin (a user), database user – and then I had to make names for them all before I understood what each one did or how many more were needed.

There was a slight snag when the wiki wanted the database password and I wanted a new one, but someone blissfully using the same password for everything wouldn’t have noticed. A less tech savvy individual wouldn’t have thought to try looking for where to copy the wiki’s configuration file via command line. Once I figured that out though, I landed on a fresh wiki.

The snag that caught me was the mission-critical “What You See is What You Get” editor. Whenever I tried saving changes I made with it, it returned “[<RANDOM HEX NUMBER>] Caught exception of type Error.” A help topic on MediaWiki.org [1] reported fixing his wiki by installing a package called php7-zlib. This package is not in the Synology-approved repository, and I found no other package managers I’m familiar with when I connected over SSH. That’s… understandable, I suppose. The product is aimed at homes and small business too small for dedicated IT, after all.

An Alternative MediaWiki Host

A couple weeks ago, I had the misfortune of breaking one of the hooks securing the bezel around my laptop’s screen. Without it, I have to be extremely careful opening and closing the lid. I’m in the market for a new laptop, but in the meantime, the machine’s mind as it were is intact and I can’t use it for computing just anywhere anymore.

I learned a lot on my first successful prototype wiki. Database-website distinction and multi-site databases come to mind as relevant to my use-case. I’m imagining a system where I run each website in an OCI container with Podman on my laptop, then they go to a database on GoldenOakLibry for content.

…Podman isn’t in the Debian 10 repositories. There is a way to install it that involves a lot of hububaloo, but https://pkgs.org/ says it is on Debian 11, and I’ve had the computer upgrade bug as of late. My recent experience upgrading Mint primed me to locate a tutorial and upgrade to Debian 10. The process was the same (Timeshift, shift repositories, upgrade), just a bit less automated [2]. I took the opportunity to clean up after a failed project or two that involved repositories, but I think I ran into issues with Lutris’s repository GPG key (it updated later so I’m not sure). I’m leaving it for now.

The packages podman, cockpit, and cockpit-podman went on easily. Getting a static IP for the laptop was another story. Its official position within the house is under the TV, out of range of any free Ethernet cables we have laying about. After a few hours trying to understand how its Wi-Fi is even connected, I chose to move it next to ButtonMash and configure a static IP that way.

I started and enabled Cockpit with systemctl. It complained without a proper config file, but a browser on another computer made it to laptop’s Cockpit login screen. I told ButtonMash to link Cockpits, and it gave me a command I’ve been looking for for years.

ssh-keyscan -t ecdsa-sha2-nistp256 localhost | ssh-keygen -lf -

Admittedly, this only hints at a formula, but I saved it to a special directory on GoldenOakLibry anyway.

My Second Operable Prototype Wiki

With a more malleable host than GoldenOakLibry that wasn’t ButtonMash, I scrapped what I could of my first setup and started over. MediaWiki lists four packages as dependencies, and I removed three of them that related directly to serving web pages. MariaDB 10 stayed because I know for sure that it is compatible.

Unlike my experience with Rocky Linux 8, Podman on Debian 11 did not come with any unqualified registries configured, so I was getting fast searches with no results when pulling an image in Cockpit. I took a break for Sabbath, even though I felt I could keep the progress coming. When I got back, I about immediately found a tutorial that recommended a couple Red Hat container registries to add in addition to docker.io [3]. I spotted registry.centos.org in ButtonMash’s registries.conf; with the warnings in the file headers about who you trust, I removed it over the slim chance it gets compromised in the future. Worst case scenerio: I have to re-add it later.

Acquiring docker.io’s official image was easy next to telling MariaDB to let it in. I spent around seven hours inching through assorted tutorials tangential to setting up MediaWiki in a Podman container with [important keywords here:] remote access to a MariaDB database on a Synology device. It was slow, I could have written a post about just this paragraph, but I learned enough to understand the provided instructions (key tutorial: [4]). I braved Vim to write a needed config file and learned about MySQL database CLI client to make a pseudo-root account. And of course this was after locking things down to the static IP addresses I set up earlier.

Once MediaWiki was happy with its access to MariaDB, setup was similar to my first time, though I paid a little closer attention this time around and included all the editors, the mistake that send me on this side quest in the first place. The containerized setup will still come in handy, so it was not all for nothing. As a final, problematic sendoff, MediaWiki’s setup file, LocalSettings.php, remembers the port number it was installed to: future wiki installation attempts will happen in the containers they’re meant to run in, not some baseline I’ll be keeping around.

It was cause for celebration when I made the first edit and it stuck.

Project Notes

Given the right circumstances, I would have to say it’s possible for about anyone to bumble his or her way into a working self-hosted wiki on a Synology NAS, as I sort of did. Don’t get me wrong: even this is not an impatient beginner’s project! This week I learned that databases stand alongside websites, not inside them – a very important distinction for a sysadmin to know.

I’ve seen the Cockpit functionality to switch hosts since first installing Rocky 8 on ButtonMash. It was a pleasant surprise to find it worked over SSH and had a ready command for generating SSH host key fingerprints. DSM sadly does not have that functionality.

My opinion of Synology’s DSM began strong after a slow start, but it’s been fading. Stray one command outside their intended use case and it has DON’T TOUCH THAT! signs waiting everywhere. It’s still production grade, and that I can respect. I just won’t be asking for a similar system in the future.

The database password was extremely difficult to get right. No errors were ever thrown when entering 100+ character jibberish from Bitwarden, but 79 appears to be the maximum MySQL can swallow.

Takeaway

My progress this project does not represent a production-ready environment. I fully expect to have to tweak things before I have each wiki sequestered to its own user while still running happily. Website administration will be a whole other matter to conquer, but that is an exercise for another week.

Final Question

What kind of information might you organize with a wiki?

Works Cited

[1] Winel10, “Caught exception of type Error when saving changes in VisualEditor,”MediaWiki.org, Feb. 4, 2019 and June 8, 2022. [Online]. Available: https://www.mediawiki.org/wiki/Topic:Uuk96xjvh0ukaci2. [Accessed June 27, 2022].

[2] AM, “How to upgrade to Debian 11 from Debian 10,” AtechTown.com, 2022. [Online]. Available: https://www.atechtown.com/upgrade-debian-10-to-debian-11/. [Accessed June 27, 2022].

[3] J. Arthur, “How to Install Podman on Debian 11,” LinOxide.com, Sept. 20, 2021. [Online]. Available: https://linoxide.com/install-podman-on-debian/. [Accessed June 27, 2022].

[4] TechNotes “How to run Mariadb in Docker and Connect Remotely,”YouTube.com, Dec. 15, 2020. [Online]. Available: https://youtu.be/OabTOPOU2RU. [Accessed June 27, 2022].

Posted on

I Switched My Operations to Caddy Web Server

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am rebuilding my home server, Button Mash, from the operating system up. Let’s get started!

Caddy Over Nginx

I spent well over a month obsessing over Nginx, so why would I start over now? As far as I am concerned, Caddy is the piece of software for my use case. While I am sure Nginx is great at what it does, I keep slamming into its learning curve – especially with integrating Let’s Encrypt, a tool for automating SSL encryption (HTTPS/the green padlock). Caddy builds that functionality in while still doing everything I wanted Nginx for.

The official Caddy install instructions [1] for Fedora, Red Hat, and CentOS systems are as follows:

$ dnf install ‘dnf-command(copr)’
$ dnf copr enable @caddy/caddy
$ dnf install caddy

First of all, new command: copr. Background research time! COPR (Cool Other Package Repositories) is a Fedora project I feel comfortable comparing to the Arch User Repository (AUR) or Personal Package Archive (PPA): it lets users make their own software repositories.

Installation went smoothly. When I enabled the repository, I had to accept a GPG key that wasn’t mentioned in the instructions at all. From a user point of view, they appear to fill a similar purpose here to a SSH keys: special numbers use math to prove you are still you in case you get lost.

Caddy uses an HTML interface (a REST API – Don’t ask, I don’t understand myself) on the computer’s internal network known as loopback or localhost on port 2019. Caddy additionally serves everything over HTTPS by default. If it cannot convince Let’s Encrypt to give it a security certificate, it will sign one itself and tell the operating system to trust it. In other words, if I were not running ButtonMash headless (without a graphical interface), I’d be able to try connecting to localhost:2019 with a favorite browser, like at least one of the limited supply of Caddy tutorials did.

IP Range Transplant

I should have just done my experimentation on DerpyChips or something. Instead, I pressed on with trying to point a family-owned domain name at Button Mash. This side adventure sprouted into last week’s post. In short: ButtonMash’s static IP kept was in conflict with what my ISP-provided equipment kept trying to assign it, resulting in an estimated 50% downtime from a confused router. Upgrading to the next gateway may have allowed us to free up the IP range for the gaming router’s use, but it’s not out for our area yet. My father and I switched our network connections over to a “gaming router” we had laying about and enabled bridge mode on the gateway to supposedly disable its router part. I have my doubts about how it’s actually implemented.

Most of our computers gladly accepted the new IP range, but GoldenOakLibry and ButtonMash –having static IP’s– were holdouts. I temporarily reactivated a few lines of configuration on my laptop to set a static IP so I could talk with them directly and manually transfer them over to the new IP range, breaking NFS shares and Vaultwarden on them respectively.

In the confusion, ButtonMash lost its DNS settings; those were easy enough to fix by copying a config line to point those requests to the router. GoldenOakLibry took a bit longer to figure out because the NFS shares themselves had to accept traffic from the new IP range with settings buried deep within the web interface. Once that was sorted, I had to adjust the .mount files in or around /etc/systemd/system on several computers. Editing note: While trying to upload, I found I could not access GoldenOakLibry on at least a couple of my machines. Note 2: I had to change the DHCP settings to the new IP range on my Raspberry Pi reverse Wi-Fi router. Systemd on both goofed systems needed a “swift kick” to fix them.

sudo systemctl start <mount-path-with-hyphens>.mount

Repairs Incomplete

That left Vaultwarden. I was already in a it’s-broken:-fix-it-properly mentality from the modem/router spinoff project. I got as far as briefly forwarding the needed ports for an incompletely configured Caddy to respond with an error message before deciding I wanted to ensure Bitwarden was locked down tightly before exposing it to the Internet. That wasn’t happening without learning Caddy’s reverse proxy, as I put Vaultwarden exclusively onto a loopback port.

Speaking of loopback, I found the official Caddy tutorials lacking. They –like many others after them– never consider a pupil with a headless server. I have not yet figured out how to properly convince my other computers to trust Caddy’s self-signed certificates and open up the administration endpoint. That will come in another post. I did get it to serve stuff over HTTP by listing IP’s as http://<LAN address>, but Bitwarden/Vaultwarden won’t let me log in on plain HTTP, even over a trusted network and confine the annoying log to a file.

As far as I can tell, the administration API on port 2019 does not serve a normal web page. Despite my efforts, the most access I have gotten to it was for it to error out with “host not allowed.” I haven’t made total sense of it yet. I recognize some of the jargon being used, but its exact mechanics are beyond me for the time being.

Takeaway

Caddy is a powerful tool. The documentation is aesthetically presented and easy enough to understand if you aren’t skimming. But you will have a much better time teaching yourself when you aren’t trying to learn it over the network like I did.

Final Question

Do you know Caddy? I can tell I’m close, but I can’t know for sure if I’m there in terms of the HTTP API and just don’t recognize it yet. I look forward hearing from you in the comments below or on my Discord server.

Works Cited

[1] “Install,” Caddy Documentation. [Online], Available: https://caddyserver.com/docs/install. [Accessed: June 6, 2022].

Posted on

Patience is a Virtue When Internet Breaks

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

It’s the last Monday of the month. I should have a larger project this week, but I had to split a step off into a prequel of sorts when it ballooned on me. For reasons I will go into next week, I ended up talking with tech support for my Internet provider.

It all started when I logged into my router’s off-site network settings (WHY??? is this a thing?) so I could forward a couple ports to Button Mash, my home server, when I noticed it alternating between two internal IP’s: the static IP I configured it to call itself and another one I thought was assigned to another [possibly decommissioned at present] Raspberry Pi – or it may have been a setting I never fully cleaned up. I have no way to tell.

Additionally, I learned that the online toy they’re having me use to try and manage the network is being retired. They want me to use their app I don’t want on the phone I don’t carry. No thanks. I’d rather move my whole network over to this 3rd party gaming router we have on hand.

Support Begins

Friday before posting: On my first session with support, I got a live agent after complaining to their help bot that it hated me. When I shared how I wanted to use my own router, the agent mentioned “bridge mode,” an option I had noted in the gateway’s admin panel. We continued by poking at my Manjaro workstation’s 10 Mbps connection speed with little luck because I was using a different computer.

Sunday: I got into chat with an agent on my now 3.5 Mbps connection. To not-my surprise, the obligatory reboot everything between me and the Internet didn’t change much. I told him no fewer than two or three times that I was on desktop, not phone; I do not have mobile data to remain connected; no, I do not think my family is interested in a coupon for this streaming service you’re offering; this is my one and only means of connecting to you. I followed my given instructions and pressed a button on the back of the gateway for an unusually long time. The Internet did not come back on.

Level 2

My father called a number given to me by the second chat agent, and we were connected with an awesome “Level 2” agent who could follow along as I described my unique home network while repairing the damage done by a factory reset. When bridge mode came up, I figured we might as well fix it how we want if it’s already broken and we have a professional on the line. He had some sudden technical issues of his own an hour plus into helping us.

We called back and got someone whom I had to remind a couple times that Debian 10 does not mean Windows 10. Nevertheless, he got the idea through to us that moving/disabling the default IP range was not a feature our gateway supports. I read something as much in a forum post; it said something about Layer 2 network devices, but that’s homework for a future topic.

The first guy called back just as I was giving into despair with the second guy. Long story short: our network is at around 80%; ButtonMash and GoldenOakLibry, our server and network storage, are configured with static IP’s and don’t show up at all right now. The awesome agent suggested upgrading to the next router, which might not be hard coded to serve its default IP range, allowing me to swap IP ranges between gateway and router. I don’t want to manually configure all my computers to find GoldenOakLibry at a new IP.

Takeaway

A lot of going through tech support as a customer is about explaining your exact situation to however many agents you may come into contact with.

Special thanks to Mr. E on my family’s personal Discord server for the suggestion of disabling the gateway’s integrated router.

Final Question

Have you ever had tech support struggling to keep up with you?

I look forward hearing your answers on in the comments below or on my Discord server.

Posted on

Installing NUT UPS Driver on Rocky Linux 8

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am installing the Network UPS Tool on my Rocky Linux 8 Button Mash server. Let’s get started!

A Package Exists

In a previous push on my Button Mash server, I talked about getting an Uninterruptible Power Supply (UPS) so ButtonMash could shut itself down in case of a power failure. If memory serves, I also talked about an open source driver called Network UPS Tools (NUT). At the time, I was under the impression it was exclusively available via source code and I would have to compile it to make it work.

I’ve recently suffered no fewer than four power outages since installing the UPS. A couple long ones while everyone in bed would have outlasted the UPS’s endurance had someone not noticed been aware each time to gracefully shut things down manually. I want the process automated.

And so I started the grind. The first thing the installation instructions tell me is to check for a package. Sign me up!

dnf search nut

I got several results, but with such a simple package name, the letters n-u-t turned up many false positives. NUT’s companion packages come with names of the form: ‘nut-*’, so I often filtered with ‘nut-’. My refined searches remained empty.

Installing EPEL and NUT

If the backbone of a distribution is its package manager, repositories would be its ribs. Not every piece of software gets compiled and packaged for every architecture/package manager. I get that. It was a lesson I had to learn last time I played with optimizing MicroCore Linux and why I’m going with Arch if there ever is a next time.

When I learned NUT was widely available in package form, I went looking again on Rocky Linus dnf: still nothing. Debian has a nice package viewer[1], so I looked for something similar for Red Hat distos. I wanted to be sure I wasn’t missing something before concluding the nonexistence of a package for me. One exists, but I’d need to make an account. However, I found something even better for my purposes.

pkgs.org[2] is a website that lists packages organized by several different major distributions. I was quickly able to find NUT in the CentOS 8 section for the Intel CPU architecture, but not anywhere under Rocky Linux.

A closer look after hours of confusion introduced me to the EPEL repository (Extra Packages for Enterprise Linux). Apparently, it’s held in high regard among the Red Hat branch. Many enterprise Linux users consider it almost mandatory to offset the smaller offering by default repositories. I was uneasy about it at first because it showed up for the now depreciated CentOS RHEL downstream, but EPEL is maintained by the Fedora community, which isn’t going anywhere for the foreseeable future: I’m calling it safe to use.

sudo dnf install epel-release
dnf search nut

NUT was then as simple to install as any other program from a repository.

Side Project

Podman pranks again! While testing my Bitwarden login from my laptop, I got myself permanently logged out. I traced the problem back to my Podman container on ButtonMash corrupting during one of those power outages from earlier. I sent a discouraging error off to the search engine and I found my exact issue on the Podman GitHub (see Works Cited) [3]. I wasn’t happy with the explanation, but it was the best one I found: systemd didn’t like an under-privileged user doing things without at least a recent login, so it messed with Vaultwarden’s Podman container. The messed up container had to be forcefully deleted and remade. I also needed to remember to specify https:// when looking for the server via browser. To make sure it doesn’t happen again, I followed a piece of advice found later in the discussion and permitted the login to linger.

Takeaway

I honestly expected this week’s progress to take at least a month. When I first looked into NUT, all I saw was source code ready to download and compile and honestly, I’m having trouble getting excited about mastering the art of compiling other peoples’ code. If there’s a way to install via a compatible repository, I’m all for it.

I am especially thankful for pkgs.org [2]. They helped me reduce my problem to one I’ve at least blindly followed a tutorial for. You typically won’t find the full, non-free version of Chrome on Linux, so when I was setting up Mint for my father, I had to explicitly add a repository.

While NUT may be installed, configuration is not happening this week if I expect to understand my system when I’m done. I blitzed the first expected month of work and only stopped because the next bit is so intimidating. Here’s to a quick understanding within the next month.

Final Question

NUT has proved difficult to locate assistance for, as I haven’t figured out how use their internal system. Do you have any idea where I can find support for when I need it?

Works Cited

[1] Debian, “Packages”Debian, July, 2019,Available: https://packages.debian.org [Accessed: Jan. 10, 2022].

[2] M. Ulianytskyi, “Packages for Linux and Unix”pkgs.org, 2009-2022, Available:https://pkgs.org/ [Accessed: Jan. 10, 2022].

[3] balamuruganravi “rootless podman ERRO[0000] error joining network namespace for container #6800” github.com, Jun 2020. Available:https://github.com/containers/podman/issues/6800 [Accessed: Jan. 10, 2022].

Posted on

ButtonMash’s Solid Foundation on Rocky Linux

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am still working on my Rocky Linux server. Let’s get started!

Project Selection

One would think it shouldn’t take a month to set up a server, but the vast bulk of that is research. What all do I want the server to do? What services do I need to set up to include it? When I know more than one way to do something, which way do I want to do it? The questions don’t end until work has progressed beyond the point of answering differently.

My goal for today is to get a few things running: I want to mount the GoldenOakLibry NFS server. I want to update-grub so I can properly dual boot with Debian. I want to install BitWarden. These three things are probably the most important end-goal tasks remaining for configuring ButtonMash’s Rocky install.

Package Managers

Before I can really work on my target goals, I need to know some of the basic specifics. Every major branch has its own compatible package managers. Debian has DPKG and Apt (Snap for the Ubuntu sub-family) while Arch has Pacman and AUR. Wrappers and cross-compatibility tools exist as a zoo of possibilities that will not be fully sorted out here, today.

My first impression as I research the Red Hat branch’s solution are the striking parallels to Debian, though it is also experiencing a stir. RPM (Redhat Package Manager) is like DPKG in that it is used for directly interfacing with the repository. YUM (Yellow dog Updater, Modified) was the package manager the likes of Apt I’ve been hearing about associated with the branch. It is now replaced by DNF (DaNdiFied YUM) for installing Package X and everything Package X needs to run (called “resolving dependencies”). Both YUM and DNF are present on my install, though.

Cockpit

I’ve had a chance to look over this web interface that came with Rocky Linux. By default, there doesn’t appear to be much to it after logging in beyond information readouts, an interactive firewall, and most importantly: an in-browser terminal. There appears to be a whole ecosystem to learn about, but it’s beyond me for now. I will want to look deeper into this subject when I move in to disable password authentication over the network.

Note about the terminal: it’s a little quirky from sharing its inputs with the browser. Nano’s save command also tells FireFox to “Open” and copy-paste commands don’t always work the same.

NFS Mount

From experience, I know that NFS is a royal pain to learn how to set up. On top of that, I know of at least two ways to automount network drives: during boot with fstab, and dynamically with systemd. Mounting it with fstab is annoying on my laptop because it halts boot for a minute and a half before giving up if GoldenOak is unreachable. More annoying is that this appears to be the more well documented method between the two. For an always-on server, though, it may not be a concern.

Not helping systemd’s case is/are the additional way/ways I’m discovering to set its automount functionality up. I don’t even know the proper name for the method I’ve used before – just that I didn’t mess with /etc/fstab whereas another systemd method does. It is a great challenge finding a source that compares more than a single mounting method. The good news is that aside from installation, I should be able to disregard what distro the tutorial was intended for.

While researching this section, I rediscovered autofs, and saw mention of still other automount methods. I’m avoiding autofs because because the more I read about it, the move complex it appears. In this instance, it would behoove me to just leave a line in /etc/fstab because I don’t expect to be booting this server outside the context of the GoldenOak NAS, but as this is more or less the centerpiece of my home’s network, I’m going with systemd mount files, as per the blog by Ray Lyon I referenced last February when I first learned about it. I’ll leave a link to his post in my Works Cited[1].

NFS Automount is tricky stuff, but each time I study it, I retain a little more. I can barely remember how to mount a share manually – let alone configure systemd automounts. It took me several days to find a copy of the files I needed, even after looking back at my above mentioned post from February[2]. My best guess is that I got lost in my own filesystem. I’m taking notes and organizing them in my home directory on this new install.

Update-Grub

When I installed Rocky Linux, I was all nice and safe by not letting it see any drives it wasn’t installing over, but the host machine still has a job to do on the photo trunk project; I need it to dual boot. I read up on a command called update-grub I could just run once everything was installed and physically reconnected. First of all, update-grub is a script, and second of all, it’s notoriously absent.

A variety of help topics exist on what command to run on RHEL instead of update-grub. From what I can tell, it’s pretty universally present on Debian-based systems and when I checked Manjaro (Arch family) just now, it was there too.

Update-grub itself is pretty simple. It’s three lines long, and serves as an easy-to-remember proxy command to actually update your Grub boot loader. The exact command may differ between computers depending on if they’re using BIOS or a newer, less common equivalent called UEFI. I assume it is generally generated during package installation.

Once I had my bearings, it was fairly easy to update grub on my own. I found my configuration file at /boot/grub2/grub.cfg because I am using BIOS. An effectively empty directory stump existed for the UEFI branch, cluing me in that this operation is one you should understand before using copy-paste into terminal. This StackExchange forum has several individual explanations, including reference to what I take to be a catch-all I am not using. Link[3]

So… I go to verify everything is working, and it’s not. A simple reboot loaded Rocky’s GRUB, but the Debian kernel refused to load over the USB 3 PCI card. So much for that idea. I moved the Debian drive to a motherboard USB port and BIOS found it and loaded Debian’s GRUB, which doesn’t know about Rocky Linux. I tried running update-grub in Debian and… it didn’t work. I wasn’t looking to spend even more time on this part of the project, so after confirming that Rocky’s GRUB could boot Debian, I got into BIOS and told them to prefer the internal Rocky drive over anything on USB.

BitWarden False Alarm

I’m super-excited about putting my self-hosted BitWarden server back up. I’ve already started researching, but the topic still feels like it’s expanding when I need to be getting ready for publishing this already lengthy post full of amazing progress. BitWarden will need to wait until I can better teach myself how to properly take care of it.

Takeaway

The Red Hat branch of Linux is in a notable state of flux. Key fundamentals elements of the family like CentOS and YUM are everywhere in old tutorials, and that is bound to make for a frustrating time trying to learn Red Hat for a while to come – especially if you’re new to Linux. Here, more than anywhere else, learning the history of the branch is vital to teaching yourself how to sysadmin.

Side Project

A while ago, I thought Derpy’s RAM was failing because Kerbal Space Program kept crashing the whole system. I’ve been running the three 4 gigabyte sticks on my Manjaro workstation for a month or two, and they appear fine. In the meantime, my father ordered up a pair of 8gb sticks. This week, I installed them, displacing one of the 4gb sticks. Passive testing will now commence.

Final Question

Have you ever had a project take a discouragingly large amount of research time then suddenly come into focus in a single day?

Works Cited

[1] R. Lyon, “On-Demand NFS and Samba Connections in Linux with Systemd Automount,” Ray Against the Machine, Oct. 7, 2020. (Edited Aug. 8, 2021). [Online]. Available: https://rayagainstthemachine.net/linux%20administration/systemd-automount/. [Accessed Nov. 7, 2021].

[2] Shadow_8472, “Stabilizing Derpy Chips at Last,” Let’s Build Robotics With Shadow8472, Feb. 22, 2021. [Online]. Available:https://letsbuildroboticswithshadow8472.com/index.php/2021/02/22/stabilizing-derpy-chips-at-last/. [Accessed Nov. 7, 2021].

[3] “Equivalent of update-grub for RHEL/Fedora/CentOS systems,”StackExchange, Aug. 26, 2014-Oct. 10, 2021 [Online]. Available:https://unix.stackexchange.com/questions/152222/equivalent-of-update-grub-for-rhel-fedora-centos-systems. [Accessed Nov. 7, 2021].

Privacy Policy