Good Morning from my Robotics Lab! This is Shadow_8472 and today is part 3 of this year’s big server push. Let’s get started!
Tuesday
Podman NFS
I had a slow start to the week, but I decided to poke at if Podman over NFS was actually as impossible as I thought. It wasn’t. A VERY special thank you to ikkeT, whom I made contact with over the official Podman Discord server, who pointed out, “nfs doesn’t know about selinux.”
Backing up a bit, I had a few minutes waiting for supper today, so I decided to start a clean attempt to mount a directory over NFS. My trials leading up to creating a persistence volume worked until I tried mounting one from GoldenOakLibry over NFS. Once I removed SELinux’s :z permissions flag per ikkeT’s advice, the container started and I found a flag I’d previously left in that directory.
Sadly, applying this quick fix did not solve my old scripts. I plan on re-building them this week.
In light of this discovery, I moved a 1TB solid state drive labeled “MineOS” from ButtonMash back to GoldenOakLibry. Around half a dozen micro-blunders later, I additionally removed its line from ButtonMash’s file system table (fstab) and noted that Debian 11 –which my RedLaptop is running– will be facing an End of Life (EoL) situation around a month after Rocky 8 on ButtonMash.
Vaultwarden Migration
It was a bit messy, but I did a “Take II” on migrating Vaultwarden into Caddy’s reach. I used root to copy the vw-data directory from Vaultwarden’s dedicated user to PiHole’s. I copied it again as PiHole’s user to correct root’s ownership of all the sub-directories, and then removed root’s copy… confirming. one. file. at. a. time.
For the migration proper (as well as my NFS testing earlier), I learned a container with a minimalistic set of command line tools called BusyBox. I spun it up mounting both a newly created Vaultwarden volume alongside the old directory. Inside, I copied all files over.
I started Valulwarden using the new volume and Caddy for the reverse-proxy to access it, but I also had to stop the production grade Pi-Hole in favor of the development configuration. I navigated over to it with my local domain and found the migration was a success.
Android root.crt Import
When I got up this morning, I told myself I would install the root certificate from Caddy on my Android tablet. I copied the file over USB, and then went to Settings>Security>OTHER SECURITY SETTINGS>CREDENTIAL STORAGE>User certificates, where a pop-up menu found it.
Certificate in-place and environment pointed at my Vaultwarden subdomain, Bitwarden now signs in with little issue.
Wednesday
Goals for Today
Still so much to do. My main target for today will be re-building Nextcloud’s script to store data over NFS so it can be hosted by either ButtonMash or RedLaptop.
Likewise, it will be good to have Pi-Hole’s data hosted in a common space to be hosted in parallel, though I’ll want to consider how the two instances will fight or get along with each other. I’ll want to compose a help request to a Pi-Hole community after I have Nextcloud working.
Minor Cleanup Tasks
Vaultwarden: I migrated my Bitwarden clients on my daily drivers (DerpyChips and my Upstairs Workstation) to the new Vaultwarden server. For Derpy, I did concede to installing Caddy’s root certificate directly into Firefox without pressing for making it access the system’s trust anchors. I may or may not have forgotten about that issue when logging out.
Pi-Hole: I spotted a configuration with the upstream DNS pointed incorrectly. It must not have migrated correctly. It’s now pointed at its local upstream router. I additionally did a little research on a redundant Pi-Hole, and I think I’ll not fuss with joint memory after all.
Unbound: This is more of scoping out a project closely related to Pi-Hole. Unlike many of the other projects I’m using, Unbound’s developer’s, Nlnet Labs, do not appear to have an official Docker container.
RedLaptop’s Package Manager
Debian needs more attention than I pay it on RedLaptop. Judging by its errors, it looks stuck on invalid cryptography signatures related to repositories for OBS, a streaming program I never got working properly. I uninstalled the suite, but had difficulty determining which repository needed to be removed – if any. Follow up investigation strongly hinted involvement with Lutris, which I want to keep. I found a page on software.opensuse.org and puzzled out which command to apply for my own setup. This did not change anything.
I poked around farther in apt’s config files. I cleared a warning by re-enabling a WINE listing and pointing it at Debian 11’s repositories.
Podman Sorely Out of Date… Maybe?
3 strikes against RedLaptop’s Podman! It doesn’t support secrets. It doesn’t support “volume export” or “volume import” for tarballing volumes (as I found out while duplicating Pi-Hole from ButtonMash just now). But one mission-critical piece is networking between containers and pods.
Previously this month, I learned about how Podman 4.0 brought in a new networking protocol. RedLaptop has Podman 3.0.1 available per Debian’s “Stability first” mentality. When I failed to find Podman’s network sub-command by tab-to-complete, my first instinct was to try installing a newer version from a Debian backports/main or backports-sloppy/main repository. I found no such package exists for installation, and found a topic on Reddit confirming my observations.
Compiling a new version myself would be an option, but could take anywhere from an hour to a full week. Maybe a package intended for Debian 12 would work, but the path of least resistance toward is learning to work with the old standard, slirip4netns. Long story short, for my purposes, I didn’t need to change anything after all, though I did passively learn a little about how Podman networks have IP address ranges same as any other subnet.
Pi-Hole and Caddy
The plan here is for redundancy for planned downtime for ButtonMash. I have an end-of-year goal of starting to scan those family photos by the end of the year.
So, continuing to work on RedLaptop. Pi-Hole is already running with copied volumes from ButtonMash. For port forwarding, I installed firewalld, used the commands from last week for port forwarding, and accidentally locked myself out when reloading the firewall. SSH was still open though, so Cockpit on ButtonMash bailed me out without needing physical access or going straight terminal. While I was at it, I opened the ports I plan on using for now.
I confirmed my work by accessing the admin panel on RedLaptop’s Pi-Hole and testing its DNS lookup with nslookup. To finish the task, I told the home router to recommend RedLaptop as a secondary DNS option over DHCP and gave the two installations in-container hostnames based on their respective host machines.
Caddy needs to be a different story. RedLaptop is getting its own domain so I expect more headache trying to work out domain names than if I only adapt my Caddyfile.
Goals
I didn’t reach either of my stated goals per-se, but that’s OK. My understanding of running redundant Pi-Holes has grown considerably and I did partially address a few issues on RedLaptop. I’ll work on Nextcloud tomorrow for sure.
Thursday
Nextcloud or Bust
It’s Nextcloud day. In sequence to get Nextcloud running on RedLaptop, I made a subdomain in PiHole, added a reverse proxy entry in Caddy, corrected the IP’s in the Caddyfile, backed up my working Nextcloud deployment, networked a Nextcloud’s pod to Caddy’s and Pi-Hole’s, and added Caddy’s root certificate to my workstation’s trust stores.
Nextcloud then got back to me, “Access through untrusted domain,” pointing me toward config/config.php. I found the window from Cockpit-Podman a bit small for such a large file, so I mounted Nextcloud’s persistency volume in a BusyBox container… no nano text editor. While BusyBox had vi, but not vim, Nextcloud has none of the above. I pulled up a vi/vim cheat sheet.
There exists a meme I once saw about using a young programmer trying to exit vim as part of a random text generator. It’s not far off. Even with the cheat sheet, it took me a few minutes – especially with vim having some extra and equally unintuitive shortcuts to save and quit. For the record, I found :w to save and :q to quit.
After changing out RedLaptop’s static IP for nextcloud.redlaptop.lan as a “trusted_domain,” I reached the login screen. I adjusted my password in Bitwarden accordingly.
While trying to add another entry to my Caddyfile, I decided to cement how I’m going to organize it: alphabetical by domain, the successive subdomains. Top level domains point at individual machines, each service gets a subdomain, and admin consoles get a subdomain from there. Panels strictly for admin access get an admin subdomain. I’ll have to think about the merits of how I assign Podman network IP’s to containers. Right now, it’s in the order I first set them up, but I’m thinking a hash might be of merit.
Next, I followed up on my work to restore the solid state drive as a GoldenOakLibry NFS share. I re-enabled the automounts on ButtonMash and copied over the required files to RedLaptop. And then I did some last-minute tests on mounting volumes over NFS. I could create volumes no problem on an NFS share, but using them suddenly requires root permissions. I launched a help request to the Podman Discord and continued with mounting directories.
Continuing on with an old plan involving three mounts, I aimed MariaDB and the general Nextcloud data volumes at the small, but fast NFS share. But where exactly to host the bulk storage? I examined the old volume with BusyBox and found that users each have their own photos. I’m totally making a dedicated Photo Trunk account tomorrow and mounting it in there.
Redis is recommended for caching data to maintain performance as your database grows larger. On a passing inspection, it didn’t look all that bad to set up, so I downloaded a container and added it to the Nextcloud pod.
Thinking ahead about final presentation of the Photo Trunk project, I checked in on the image board I thought I wanted, Philomina, only to learn many such projects rely on Amazon Web Service, which is a hard pass for my open source, self-hosted approach.
Goals
I’d say I did a lot better, but I still have a bunch of cleanup to do. Tomorrow, I want to make this new Nextcloud installation, create accounts for admin, myself, and a special one dedicated for Photo Trunk.
Friday
I started work by navigating to nextcloud.redlaptop.lan and found it totally blank. The pod wasn’t running. Whatever I did after correcting a capitalization mismatch, Podman locked up on me. I rebooted. While Pi-Hole and Caddy went up manually, I found the Nextcloud pod spazzing out. From MariaDB’s container, one representative repetition of many:
2023-10-27 22:10:57+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
chown: changing ownership of '/var/lib/mysql/': Operation not permittedNextcloud’s container was similarly complaining about blocks of chown permissions not working. I reached out for help again, and was recommended to look into root squashing. I tried a couple different things, updated GoldenOakLibry, and still got the same result. Further investigation will be required.
Goals
It grows close to sundown, so I must break it off here. My goal for Saturday night/Sunday is again Nextcloud over NFS. I’ve been having a lot of grief out of Podman locking up on me, so I might have to reset the whole thing. I’ve had to do it before, but with everything in the same place this time, I’ll want to write a script to pull all container images I use.
Saturday Night
New Automation Scripts
Podaman is still giving me issues with containers stuck in improper states. I prepared a script to re-download all my container images so I don’t have to rebuild manually in case this keeps happening. While this is a workaround, it may allow me to notice a pattern if I observe Podman failing enough times in the long run. Right now though, the only pattern I see is that rebooting semi-fixes Podman hanging for a few minutes of me working on it.
I also created a script to call each of the other startup scripts. I don’t anticipate needing it in the long term though. One thing of note for both this and the reload script is that I wrote it from my tablet, which has previously refused to load Cockpit through the browser I was using.
Podman Reset and Cleanup
STUPID! While going to do the heavy duty Podman reset, I specifically checked its warning:
$ podman system reset
WARNING! This will remove:
- all containers
- all pods
- all images
- all build cache
Are you sure you want to continue? [y/N]NO MENTION OF VOLUMES!!! It nuked my volumes along with everything else. For what it’s worth, I can still rebuild relatively quickly. I’ll just design my system to work with “bind mounts” instead of relying so heavily on volumes this time around. I was NOT wanting a wipe this through!
I took this opportunity in the rough to polish a few of my file names. My biggest loss besides my (fallback Nextcloud installation) was the RedLaptop copy of Pi-Hole, which I’d modified a bit from its original. On my upstairs workstation at least, I also updated RedLaptop’s root certificate authority certificate.
Goals
A few suggestions to try came in over Sabbath. My goal for tomorrow is to look into them if nothing else.
Sunday
I spent my mental energy on an event today, but I got a little research in nonetheless. To recap: rootless Podman does things with namespaces which NFS doesn’t support.
GitHub user rhatdan, a maintainer on Podman, speculated I could use something called fuse file system. I don’t get the full concept, but it appears to be the better option should it develop.
Another GitHub user going by eriksjolund separately suggested manually managing my namespace and helped me clear out a fair number of errors. It took a few tries, but as of writing, I suspect this course of action may prove a dry end because it appears to conflict with how I’m interfacing my containers with Caddy. This suggestion too needs additional time to develop, but develop it sure did when I hit refresh and found a fresh reply. It was interesting –to say the least– working out what some of the recommended changes to my script did for myself.
This just in on the way to publication: it looks like manual UID/GID mapping has run its course unless someone new has a brilliant idea that’s simpler than compiling Podman from source or updating Debian.
To be honest, I’m getting burned out on server. It’s been an awesome run seeing things start coming to life, but I feel the need some time off from it. I’ll follow up with these two leads, but don’t be surprised if I do a Re-Learning Linux post next week.
Takeaway
I’m beginning to have moments where I see ButtonMash, RedLaptop, and GoldenOakLibry as parts of a greater whole. While working out the earlier two to operate in parallel, I’m creating a system with redundancy built in. I can work on one and improve the other when I’m confident in my abilities to do it correctly.
Final Question
Rootless Podman NFS. How?
Work Cited
[1] Shadow8472, D. Walsh “rhatdan,” E. Sjölund “eriksjolund,”“Rootless NFS Volume Permissions: What am I doing wrong with my Nextcloud/MaraiDB/Redis pod?,” github.com, Oct. 27, 2023-.[Online]. Available: https://github.com/containers/podman/discussions/20519. [Accessed Oct. 30, 2023].