Tag: homelab

Posted on

My First Computer “Rack”

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

So far, I’ve been assembling my servers (ButtonMash, RedLaptop, and GoldenOakLibry) on and under a foldable table. Add a workstation, and it’s getting a bit cluttered. We’ve had a set of glass shelves going unused for a while now, and I think they might do nicely to organize the servers’ room.

I started by measuring ButtonMash’s case against the shelves’ metal frame. While it was close, I estimated an inch vertical clearance once the shelves were in place. Otherwise, the tentative plan was to remove a shelf. My father and I moved the shelves in and loaded them and the setup’s UPS (Uninterruptible Power Supply). Wiring was relatively straightforward with the traditional wire Medusa in the back, but out from underfoot.

Unfortunately, I left ButtonMash in a precarious state such that a reboot before moving it knocked out my known house of cards supporting PiHole and Unbound. What I didn’t realize was that I never got Caddy working on that machine in the first place. In trying to fix Caddy, I wiped the containers I actually the whole house was using for DNS. As a patch, I pointed the router back at our normal DNS servers.

While I’m trying to avoid server work this month, I went ahead and looked up how to change my specific DNS settings temporarily to restart my DNS containers. From there, I did not encounter any notable issues, though I wasn’t up to testing the removal of my patch.

Takeaway

I have a rack. That’s my story and I’m sticking to it.

Final Question

How do you organize your tech stack?

Posted on

Rocky Server Stack Deep Dive: 2023 Part 5

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am learning more about Podman Quadlets for my homelab. Let’s get started!

Systemd and Quadlets

From my incomplete research going into this topic, I already know Quadlets is a system for efficiently integrating Podman containers in with Systemd. It was merged into Podman v4.4, and I had a small pain of a time trying to find a distribution with both that and legacy BIOS support along with a list of other requirements.

But what is Systemd? In short: Systemd is the init process –a process that manages other processes– used by most Linux distributions that aren’t trying to optimize for a low RAM or storage footprint. As it turns out, I’ve already had minimal exposure to it while writing unit files for NFS [auto]mounts and a static IP address on Debian. Systemd in turn bases units off these unit files to manage the operating system.


While Systemd unit files defining Podman containers can be written by hand, Quadlets can automate their creation based off simpler unit files of its own: .container, .network, .volume, and .kube. The first three look similar enough to concepts I’m familiar enough with that I figure I could hack an example into doing what I need.

But I’m interested in pods. With .pod unit files only a controversial feature request at best, that leaves me to explore .kube files, which run Kubernetes YAML files. I know nothing about writing Kubernetes YAML files from scratch, and I refuse to cram for them Thanksgiving week.

My project died here for a few hours. One Systemd tutorial brought up Syncthing in an example, and I spent a while on a tangent looking at that, but it too is too large to cram for this week. I unenthusiastically browsed back to Kubernetes, and found:

podman generate kube

Looks like I just might get away with adapting my scripts after all this week. With this in mind, I copied over my files from my laptop’s Debian drive to its new-last-week Rocky 9 installation. Focusing on Nextcloud, I cleared out my dead-end work with Fuse, abstracted volumes, and other junk before realizing BusyBox was likely a more suitable testing grounds.

My First Kuberneties File

I came up with the following bash script for such a pod:

podman pod stop busyBoxPod
podman pod rm busyBoxPod
podman pod create busyBoxPod
podman create \
    --pod busyBoxPod \
    --name BusyBox \
    --volume fastvolume:/root/disk \
    -it \
    --rm \
    busybox

And here is

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.6.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-11-23T01:29:45Z"
  labels:
    app: busyBoxPod
  name: busyBoxPod
spec:
  containers:
  - image: docker.io/library/busybox:latest
    name: BusyBox
    stdin: true
    tty: true
    volumeMounts:
    - mountPath: /root/disk
      name: fastvolume-pvc
  volumes:
  - name: fastvolume-pvc
    persistentVolumeClaim:
      claimName: fastvolume

I saved this output as busyBoxPod.yml and returned to Nextcloud.

Nextcloud put up a small tantrum getting re-updated for Podman 4.6.1. I had to look up how to Podman Secrets, and apply :z to volumes to satisfy SELinux. Redis however, refused to accept a password from Podman Secrets, so I rolled back that change. The pod should insulate it anyway. I got it to a point where it needed a domain name.

Branching out to bring up Pi-Hole and Caddy, I learned how the default Unbound configuration for the container I used only forwards DNS requests to Cloudflare. I’ll want to fix this later. I used firewall-cmd to forward ports for HTTP, HTTPS, and DNS to underprivileged ports for rootless containers.

Takeaway

UNCLE! I find more and more of my time supposedly working on server is procrastinating and stressing over either minutia or blankly staring at my screens when I muster enough focus to ignore distractions. There’s no way around it; I’m officially burned out on this project. I’ll maybe come back to it after the new year. I really wanted to get my .kube files working for at least Pi-Hole and Caddy, but it’s going to be a hard pass at the moment.

Final Question

I’m considering covering a free/open source game or few over December. What are your recommendations?

I look forward to hearing from you on my Socials!

Posted on

Rocky Server Stack Deep Dive: 2023 Part 4

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am exploring fuse-overlayfs as a possible patch between Podman and NFS. Last week’s post was practically a freebee, but I expect this one to be a doozy if it’s even possible. Let’s get started!

Context

For my homelab, I want to run Nextcloud in a rootless Podman 3.0.1 container with storage volumes on our NFS. For logistical reasons, Nextcloud needs to be on RedLaptop running Debian 11 (Linux kernel 5.10.0-26-amd64 x86_64). The NFS share I wish to share is mounted via systemd.

My most promising lead is from Podman Github maintainer rhatdan on October 28, 2023, where he made a comment about “fuse file system,” asking his colleague, @giuseppe, for thoughts to which there has been no reply as of the afternoon of November 10 [1]. I documented a number of major milestones there, which I’ll be covering here.

File System Overlays

Fuse file system turned out to be fuse-overlayfs, one of a few systems for fusing file systems. Basically: there are times when it’s useful to view two or more file systems at once. File system overlays can designate a lower file system and an upper file system. Any changes (file creation, deletion, movement, etc.) in this combined file system manifest in the upper file system, leaving the lower file system[s] alone.

Through a lot of trial and error, I set up a lower directory, an upper directory, a work directory, and a mountpoint. My upper directory and work directory had to be on the NFS, but I ran into an error about setting times. I double checked that there were no major problems related to Daylight Savings Time ending, but wasn’t able to clear the error. I sent out some extra help requests, but got no replies (Sunday, Nov. 12). A third of my search results are in Chinese, and the others are either not applicable or locked behind a paywall. Unless something changes, I’m stuck.

Quadlets

Github user eriksjolund got back to me with another idea: quadlets [1]. Using this project merged into Podman 4.4 and above, he demonstrated a Nextcloud/MariaDB/Redis/Nginx setup that saves all files as the underprivileged user running the containers. In theory, this sidesteps the NFS incompatibilities I’ve been experiencing all together.

The first drawback from my perspective is that I need to re-define all my containers as systemd services, which is something I’ve admittedly been meaning to do anyway. A second is again that this is a feature merged into Podman much later than what I’m working with. Unless I care to go digging through the Podman GitHub myself, I’m stuck with old code people will be reluctant to support.

Distro Hunt

Why am I even using Debian still? What is its core purpose? Stability. Debian’s philosophy is to provide proven software with few or no surprises left and the user polishes it to taste. As my own sysadmin, I can afford a little downtime. I don’t need the stability of a distro supporting the most diverse Linux family tree. Besides, this isn’t the first time community support has suggested features in the future of my installation’s code base. Promising solutions end in broken links. RAM is becoming a concern. Apt package manager has proven more needy than I’d care to babysit. If I am to be honest with myself, it’s time to start sunsetting Debian on this system and find something more up-to-date for RedLaptop. I’ll keep it around for now just in case.

My first choice was Fedora to get to know the RedHat family better. Fedora 39 CoreOS looked perfect for its focus on containers, but it looks like it will require a week or two to configure and might not agree with installing non-containerized software. Fedora 39 Server was more feature complete, but didn’t load up for my BIOS (as opposed to the new standard of UEFI); I later learned that new BIOS-based installations were dropped on or around Fedora 37.

I carefully considered other distributions with the aid of pkgs.org. Debian/Ubuntu family repositories go up to 4.3. Alpine Linux lacks systemd. Souls Linux is for desktops. OpenSuse Tumbleweed comes with warnings about being prepared to compile kernel modules. Arch is… Arch.

Fresh Linux Installation

With time running out in the week, I decided to forgo sampling new distros and went with minimal Rocky 9. Installation went as best can be expected. I added/configured cockpit, podman, cockpit-podman, nfs-utils, and nano. I added a podmanuser account, set it up to allow-lingering, and downloaded the container images I plan on working with on this machine: PiHole, Unbound; Caddy; Nextcloud, Redis, MariaDB; busybox.

Takeaway

I write this section on Friday afternoon, and I doubt I have enough time remaining to properly learn Quadlets and rebuild my stack, so I’m going to cut it off here. From what I’ve gathered already, Quadlets mostly uses Systemd unit files, a format I’ve apparently worked with before, but also needs Kubernetes syntax to define pods. I don’t know a thing about using Kubernetes. If nothing else, perhaps this endeavor will prepare me for a larger project where larger scale container orchestration is needed.

Final Question

Do you know of a way I might have interfaced Podman 3 with NFS? Did I look in the wrong places for help (Debian forums, perhaps)?

I look forward to hearing from you on my Socials!

Work Cited

[1]. Shadow_8472, D. “rhdan” Walsh, E. Sjölund, “Rootless NFS Volume Permissions: What am I doing wrong with my Nextcloud/MaraiDB/Redis pod? #20519,” github.com, Oct. 27, 2023-Nov. 10, 2023. [Online]. Available: https://github.com/containers/podman/discussions/20519#discussioncomment-7410665. [Accessed Nov. 12, 2023].

Posted on

My Homelab Report, 2023

Good Morning from my Robotics Lab! This is Shadow_8472 with a snapshot of my different computers as of mid-March, 2023. Let’s get started!

I have felt a need for a page detailing my so-called “homelab” detailing the computers and other equipment I personally maintain. I didn’t finish any big projects this week, so here’s a first draft of that page. It’s listed in roughly the order I took ownership/responsibility for each piece. I’ve tried to gather each computer’s base model/modified configuration plus a list of its presently bootable operating systems. Warning: unexplained technobabble ahead.

Red Laptop

Dell Inspiron N7110
1 TB SSD

Debian 11

As the oldest operational computer in my fleet, this laptop has been with me for almost 12 years, though most every part was replaced while it was under extended warranty. Shortly after I started this blog, I installed Debian 10 on a USB external drive. I nuked its Windows drive one week by formatting the wrong disk, and later replaced it with a 1 TB SSD, cloned Debian inside, and it served as a get-me-online machine until the screen bezel cracked, threatening to shatter the screen the next time I’m not careful. It’s now serving as a backup home server, and it’s desktop environment has proven vital to rebounding from at least one network emergency since.

Upstairs Workstation

Custom Build

Intel Core i7-3770K (3.50GHz)
NVIDIA GeForce GTX 970 (rev a1)
24 GB RAM (2×8+2×4)

Windows 10
Manjaro
EndeavourOS (WIP)

“Upstairs Workstation” also predates when I started naming my computers. I got it for university when laptop turned up as bit underpowered. I went from Windows 7 to Windows 10 with this machine, narrowly avoiding an online tech support scam (10 didn’t support a Wi-Fi cart I was using at the time and later finished burning out when I put it back in after switching to Linux).

Derpy Chips

Custom Build
Intel Core i7-2600K (3.40GHz)
16 GB RAM

PopOS

Derpy was named for its tendency to give power kernel errors during its first service life until its retirement. When I adopted it, I gave it a new hard drive and inadverantly diagnosed the one lowballed part: its 2 TB HDD. I’ve also had issues with a stick of RAM from it, but I’ve had it running smoothly for a few years now as a secondary workstation.

ButtonMash

Dell Optiplex 7010
16 GB RAM
1×60 GB SSD
1×1 TB SSD (external)

Debian 11
Rocky 8

ButtonMash has been my main x86 experiment machine as well as my home server. I bought it from a church member who was upgrading his dental office computers. I My was host to my first Linux install: Ubuntu, but I’ve put it through a parade of distros including MicroCore (where I learned the command line), MineOS, Debian (originally from “red laptop”), and finally Rocky. Today, it serves as my primary home server, where I am setting up several Podman containers including Vaultwarden, PiHole, Nextcloud, and Minetest – each on a separate account for further isolation.

If and when we need to scan pictures, ButtonMash is dual booted with an external SSD I originally used with my Red Laptop.

GoldenOakLibry

Synology NAS

4x7TB HDD RAID 5 share
1 TB SSD share (external)

GoldenOakLibry is our home NAS intended for picture scanning. If nothing else, it’s exposed me to using a computer that’s meant to do nothing but serve files. I have a long way to go before I master it.

Raspberry Pi’s

1xRpi3B+
2xRpi4B
1xRpi400

My collection of super-flexible Raspberry Pi computers has served me well. At the moment, the Pi 4’s are serving as OpenWRT router firmware configured as Wi-Fi-to-Ethernet adapters for my upstairs workstation and my sister’s desktop. My Pi 3B+ has been inactive since its 3D printed PacMan ghost case got smashed, and my Pi 400 shares a monitor at my upstairs workstation.

Other/Misc computers

3D printer

Maker Select Plus
Added braces+feet

I am no master at 3D printing, but I know enough of the basics to scrape together most everything I try. At present, my printer needs a new fan and a printbed that isn’t warped.

Pine Phone

UBPorts Edition

The average smartphone these days spies on you. Its parent company is allowed to by the terms of service you need to accept before use. I resolved to not blindly accept any new EULA’s without knowing what I’m getting into and learning enough to be confident I’ve done something to mitigate the worst of these violations of privacy. As a consequence, I don’t have a usable phone ever since we switched providers a while back.

UPS’s

I have three uninterruptible power supplies directly under my supervision – one serving the network closet, one serving my upstairs workstation, and the other ButtonMash’s area. As of yet, I have not figured out the open source NUT driver.

My Father’s workstation

Dell Optiplex 7020
15 GB RAM

Linux Mint 20.3
Debian 11

15 GB RAM

After I bought ButtonMash, I got another used workstation out of the same dental office, replaced Windows, and gave it both sets of RAM. I later talked my father through installing Debian 11 on ButtonMash’s original hard disk

Ventoy USB (32 GB)

Not technically a computer, but easily one of my most versatile tools. Ventoy makes Linux media creation as simple as dragging and dropping a .ISO. Even better is that I can still put normal files on it!

Takeaway

Each part of my lab serves its function. There’s a certain joy when I get something new working and get to show it off.

Final Question

Which of your projects are you most proud of?

Privacy Policy