Rocky Server Stack Deep Dive: 2023 Part 5

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am learning more about Podman Quadlets for my homelab. Let’s get started!

Systemd and Quadlets

From my incomplete research going into this topic, I already know Quadlets is a system for efficiently integrating Podman containers in with Systemd. It was merged into Podman v4.4, and I had a small pain of a time trying to find a distribution with both that and legacy BIOS support along with a list of other requirements.

But what is Systemd? In short: Systemd is the init process –a process that manages other processes– used by most Linux distributions that aren’t trying to optimize for a low RAM or storage footprint. As it turns out, I’ve already had minimal exposure to it while writing unit files for NFS [auto]mounts and a static IP address on Debian. Systemd in turn bases units off these unit files to manage the operating system.


While Systemd unit files defining Podman containers can be written by hand, Quadlets can automate their creation based off simpler unit files of its own: .container, .network, .volume, and .kube. The first three look similar enough to concepts I’m familiar enough with that I figure I could hack an example into doing what I need.

But I’m interested in pods. With .pod unit files only a controversial feature request at best, that leaves me to explore .kube files, which run Kubernetes YAML files. I know nothing about writing Kubernetes YAML files from scratch, and I refuse to cram for them Thanksgiving week.

My project died here for a few hours. One Systemd tutorial brought up Syncthing in an example, and I spent a while on a tangent looking at that, but it too is too large to cram for this week. I unenthusiastically browsed back to Kubernetes, and found:

podman generate kube

Looks like I just might get away with adapting my scripts after all this week. With this in mind, I copied over my files from my laptop’s Debian drive to its new-last-week Rocky 9 installation. Focusing on Nextcloud, I cleared out my dead-end work with Fuse, abstracted volumes, and other junk before realizing BusyBox was likely a more suitable testing grounds.

My First Kuberneties File

I came up with the following bash script for such a pod:

podman pod stop busyBoxPod
podman pod rm busyBoxPod
podman pod create busyBoxPod
podman create \
--pod busyBoxPod \
--name BusyBox \
--volume fastvolume:/root/disk \
-it \
--rm \
busybox

And here is

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.6.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-11-23T01:29:45Z"
  labels:
    app: busyBoxPod
  name: busyBoxPod
spec:
  containers:
  - image: docker.io/library/busybox:latest
    name: BusyBox
    stdin: true
    tty: true
    volumeMounts:
    - mountPath: /root/disk
      name: fastvolume-pvc
  volumes:
  - name: fastvolume-pvc
    persistentVolumeClaim:
      claimName: fastvolume

I saved this output as busyBoxPod.yml and returned to Nextcloud.

Nextcloud put up a small tantrum getting re-updated for Podman 4.6.1. I had to look up how to Podman Secrets, and apply :z to volumes to satisfy SELinux. Redis however, refused to accept a password from Podman Secrets, so I rolled back that change. The pod should insulate it anyway. I got it to a point where it needed a domain name.

Branching out to bring up Pi-Hole and Caddy, I learned how the default Unbound configuration for the container I used only forwards DNS requests to Cloudflare. I’ll want to fix this later. I used firewall-cmd to forward ports for HTTP, HTTPS, and DNS to underprivileged ports for rootless containers.

Takeaway

UNCLE! I find more and more of my time supposedly working on server is procrastinating and stressing over either minutia or blankly staring at my screens when I muster enough focus to ignore distractions. There’s no way around it; I’m officially burned out on this project. I’ll maybe come back to it after the new year. I really wanted to get my .kube files working for at least Pi-Hole and Caddy, but it’s going to be a hard pass at the moment.

Final Question

I’m considering covering a free/open source game or few over December. What are your recommendations?

I look forward to hearing from you on my Socials!

Rocky Server Stack Deep Dive: 2023 Part 4

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am exploring fuse-overlayfs as a possible patch between Podman and NFS. Last week’s post was practically a freebee, but I expect this one to be a doozy if it’s even possible. Let’s get started!

Context

For my homelab, I want to run Nextcloud in a rootless Podman 3.0.1 container with storage volumes on our NFS. For logistical reasons, Nextcloud needs to be on RedLaptop running Debian 11 (Linux kernel 5.10.0-26-amd64 x86_64). The NFS share I wish to share is mounted via systemd.

My most promising lead is from Podman Github maintainer rhatdan on October 28, 2023, where he made a comment about “fuse file system,” asking his colleague, @giuseppe, for thoughts to which there has been no reply as of the afternoon of November 10 [1]. I documented a number of major milestones there, which I’ll be covering here.

File System Overlays

Fuse file system turned out to be fuse-overlayfs, one of a few systems for fusing file systems. Basically: there are times when it’s useful to view two or more file systems at once. File system overlays can designate a lower file system and an upper file system. Any changes (file creation, deletion, movement, etc.) in this combined file system manifest in the upper file system, leaving the lower file system[s] alone.

Through a lot of trial and error, I set up a lower directory, an upper directory, a work directory, and a mountpoint. My upper directory and work directory had to be on the NFS, but I ran into an error about setting times. I double checked that there were no major problems related to Daylight Savings Time ending, but wasn’t able to clear the error. I sent out some extra help requests, but got no replies (Sunday, Nov. 12). A third of my search results are in Chinese, and the others are either not applicable or locked behind a paywall. Unless something changes, I’m stuck.

Quadlets

Github user eriksjolund got back to me with another idea: quadlets [1]. Using this project merged into Podman 4.4 and above, he demonstrated a Nextcloud/MariaDB/Redis/Nginx setup that saves all files as the underprivileged user running the containers. In theory, this sidesteps the NFS incompatibilities I’ve been experiencing all together.

The first drawback from my perspective is that I need to re-define all my containers as systemd services, which is something I’ve admittedly been meaning to do anyway. A second is again that this is a feature merged into Podman much later than what I’m working with. Unless I care to go digging through the Podman GitHub myself, I’m stuck with old code people will be reluctant to support.

Distro Hunt

Why am I even using Debian still? What is its core purpose? Stability. Debian’s philosophy is to provide proven software with few or no surprises left and the user polishes it to taste. As my own sysadmin, I can afford a little downtime. I don’t need the stability of a distro supporting the most diverse Linux family tree. Besides, this isn’t the first time community support has suggested features in the future of my installation’s code base. Promising solutions end in broken links. RAM is becoming a concern. Apt package manager has proven more needy than I’d care to babysit. If I am to be honest with myself, it’s time to start sunsetting Debian on this system and find something more up-to-date for RedLaptop. I’ll keep it around for now just in case.

My first choice was Fedora to get to know the RedHat family better. Fedora 39 CoreOS looked perfect for its focus on containers, but it looks like it will require a week or two to configure and might not agree with installing non-containerized software. Fedora 39 Server was more feature complete, but didn’t load up for my BIOS (as opposed to the new standard of UEFI); I later learned that new BIOS-based installations were dropped on or around Fedora 37.

I carefully considered other distributions with the aid of pkgs.org. Debian/Ubuntu family repositories go up to 4.3. Alpine Linux lacks systemd. Souls Linux is for desktops. OpenSuse Tumbleweed comes with warnings about being prepared to compile kernel modules. Arch is… Arch.

Fresh Linux Installation

With time running out in the week, I decided to forgo sampling new distros and went with minimal Rocky 9. Installation went as best can be expected. I added/configured cockpit, podman, cockpit-podman, nfs-utils, and nano. I added a podmanuser account, set it up to allow-lingering, and downloaded the container images I plan on working with on this machine: PiHole, Unbound; Caddy; Nextcloud, Redis, MariaDB; busybox.

Takeaway

I write this section on Friday afternoon, and I doubt I have enough time remaining to properly learn Quadlets and rebuild my stack, so I’m going to cut it off here. From what I’ve gathered already, Quadlets mostly uses Systemd unit files, a format I’ve apparently worked with before, but also needs Kubernetes syntax to define pods. I don’t know a thing about using Kubernetes. If nothing else, perhaps this endeavor will prepare me for a larger project where larger scale container orchestration is needed.

Final Question

Do you know of a way I might have interfaced Podman 3 with NFS? Did I look in the wrong places for help (Debian forums, perhaps)?

I look forward to hearing from you on my Socials!

Work Cited

[1]. Shadow_8472, D. “rhdan” Walsh, E. Sjölund, “Rootless NFS Volume Permissions: What am I doing wrong with my Nextcloud/MaraiDB/Redis pod? #20519,” github.com, Oct. 27, 2023-Nov. 10, 2023. [Online]. Available: https://github.com/containers/podman/discussions/20519#discussioncomment-7410665. [Accessed Nov. 12, 2023].

Rocky Server Stack Deep Dive: 2023 Part 3.1

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project on my home server. Let’s get started!

Not My Problem

In the greater context of setting up Pi-Hole (network ad blocker) on my home server, ButtonMash, I’ve learned a thing or two about how the Domain Name Service (DNS) works and what happens when I break it locally. Normally, when a device connects to a network, a DHCP server (often on the router) advertises a DNS server. When resolving a URL, this DNS server either has the answer cached or ask another DNS server until a match is found or the system gives up.

My Internet Service Provider (ISP) has been having some trouble with its site (including our main e-mail). Not once, but twice my family asked if I was doing something with the Internet. Both times, I used a terminal application called “traceroute” to display [most of] the hops my requests went. Ones handled by ButtonMash were very short – (I tested buttonmash.lan and a known entry on my blocklist), while others took up to 30 hops. My ISP’s site fell in the later category.

However: one cell phone in the family was still reaching our ISP’s site while on cell data. This meant that the site was fine, but the failure was with the larger DNS system (or most of their servers were down behind their reverse proxy, but I thought of that later). In either case, I looked at a couple “Is it down?” type sites, and concluded the outage was most certainly not my problem.

Unbound

But I had a project to try. Unbound is a tool for increasing digital privacy by setting up a recursive DNS server. Every domain is registered at its authoritative DNS server. When Unbound receives a request, it finds the domain’s authoritative DNS server and caches it for later. This reduces digital footprints in DNS server logs, making you harder to track and reducing your vulnerability to a hacked/confused DNS servers.

I’ve been interested in building my own Unbound OCI “Docker” container for a while as there’s no officially maintained one, but I went ahead and downloaded an image from docker.io based on the quality of documentation. I spun up a container in Podman and pointed Pi-Hole at it. It worked first try with no special configuration.

It just so happened that when I brought the fix online, we were on a call with tech support, and I was able to pass my diagnosis back to our ISP to help them restore service to their customers in my community – however large or small that number may be.

Takeaway

What’s with no persistence volumes on this container? If it resets, it will have to start over on caching. If/when I come back in a few months, I may take a closer look. Otherwise, this has been a big shortcut I can live with.

Final Question

Have you worked with Unbound before? Would it even benefit from a persistence volume?

I look forward to hearing from you on my Socials!

How I would Relearn Linux #5: Basic Scripting

Good Morning from my Robotics Lab! This is Shadow_8472 with another tip for how I would relearn Linux. Let’s get started.

Scripts

No serious system administrator has time to memorize long chains of commands taking minutes to type and might need fixing after the first try. When faced with a periodic task involving a long, involved [set of] command[s], it’s preferable to write them in a script to be run sequentially from file. While I did get by for a few years by using Bash’s (terminal shell, see section on shells) history functionality, a script is a much more sustainable way to “remember” commands.

To write a script, put your command[s] into a text document – one per line. The pound sign # at the start of a line turns it into a comment to clue future you or others into your thought process. Using a backslash \ character and additional whitespace for alignment, lines may be broken up at any point to increase readability. This becomes especially useful when dealing with half a dozen flags.

#this line is a comment
podman run \
    --name testpilot \
    --network podman \
    --ip 10.88.2.88 \
    -v ~/sandbox/testVolume:/root/vol1 \
    --rm \
    busybox:latest
#No interrupting commands – even with commented flags.
#    -v ~/sandbox/wrongTestVolume:/root/vol2 \

A script will then need permission to execute.

chmod +x <filename>

Lacking a proper explanation of the Linux permissions system, just know this is a fast way to ensure YOU have permission to EXECUTE your script.

Even understanding up to this point unlocks a powerhouse of possibility. These tools –sequential commands, comments, line breaks, and white space– are all I need to make scripts to stitch together more complicated programs I work with to build my home server. Most importantly: scripting commands allows me to clearly visualize the command I’m working on.

Shells

One slightly mind bending concept is the shell. In simplified terms, a computer shell is an operating environment. Open a terminal on a Linux workstation, and you probably have a Bash shell. Some part of desktop environment you opened that shell from is a graphical shell. From Bash, you can open another instance of Bash inside the last one like a Russian matrushka doll (nesting dolls). Connect to another computer with SSH, and there’s another shell (one one each machine working together, I think?). Other programs –such as a Python interpreter– may have a shell of their own.

Not all shells are intended for human interaction. Somewhere below the graphical environment is logically a shell either running directly atop the kernel (I don’t actually know if the kernel itself counts as a shell) optimized for running quickly.

Most important to this conversation: running a script will open a shell for its own use and close it afterwords. Fancier shell scripts can leverage many powers characteristic of typical programming languages. Flow control allows logic to branch and loop depending on the state of a whole assortment of variable types. And thanks to shell manipulation, variables can be isolated from one another in ways I’m totally unfamiliar with. And all this exposed power comes packaged directly into the operating system itself.

Takeaway

Shell scripting is a powerful tool. This single lesson is enough to unlock a wealth of potential, yet only a fraction of its total capabilities. Variables –for example– turned out more involved than I thought, so I scrapped a section them after writing about shells to support it. While it’s OK to run with only a partial knowledge, it’s also good to be aware of additional capacities when reading scripts written, polished, and published for use by other people.

Final Question

What projects have you designed with scripts?