Tag: homelab

Posted on

QNAP Customer Support Failure (Unsupported Project)!

Good Morning from my Robotics Lab! This is Shadow_8472 and this month’s check in is… well… Let’s just get started.

Support Letdown

Last month, I was about a week into a QNAP support ticket to configure LDAP to an LLDAP server. Long story short: They took another three weeks to drop me. Practically a whole month wasted getting nothing out of them at all. Nothing.

The guy I talked to was polite though.

Backing up to the beginning for a recap: QNAP’s support set an expectation early that I would hear back once a day around 5:00 PM. I flubbed working with their interface, made two tickets, and they were merged without a note, leaving my support agent confused, and we were unproductive for a full day. By gathering everything I knew, I got my device logging in, but QTS (QNAP’s operating system for Network Attached Storages, or NAS’s) never populated the Domain Users list.

Using a combination of Wireshark, LLDAP’s logs, and analysis by Shimmering Dust, my Hermes AI agent, I was left with the conclusion that QTS wanted more of the LDAP standard implemented than LLDAP supports by default. My support question ended up in the back room, where I hung in limbo for a week.

Shortly after I asked if the ticket was still alive, the back room got back to my original assigned agent saying something along the lines of “LLDAP does not support SAMBA.” I don’t want SAMBA. That is Windows’ Active Directory stuff. I run in the Linux ecosystem. I saw a reference to ChatGPT in a link the back room tried to direct me to about SAMBA, and I started to wonder if it was human miscommunication or if I was subject to a third-hand AI hallucination. Turns out some of them use it as a translator from Mandarin to English (the support center is in Taiwan).

In either case, I was in-fact dealing with a human, and we had a human interaction waiting for the back room, who proceeded to advise that QTS did not support direct LDAP despite having LDAP as an option in their interface! It took a couple more rejections before I was convinced they finally weren’t under the illusion that I was about to subject my homelab to a Microsoft license agreement.

And there I was dropped. There was a little corporate ceremony about making a feature request to support LLDAP and thanking me for my cooperation and debug information, but at the end of the day, I felt like I was handled like a good-hearted, but tech-illiterate granny; my demonstrated skill level appears to have had no room for recognition in their program. I literally could have had a faster and equally productive experience explaining my problem to a rubber duck. I wasn’t even graced with a hint at which of the LDAP data fields were mission-critical for QTS to recognize a server.

I feel burned… but at least my agent was polite.

Studying for CCNA

On a totally unrelated note, I have enrolled in Network Chuck’s “Fall into CCNA” class. I’ve been meaning to get a certification or few (or at least the complete knowledge set to back them up), and decided to sign up. It takes a lot of my available focus, leaving less time for blog projects. The day this goes live is the beginning of week 3. One look through the study plan, and I either know or at least recognize most of the concepts mentioned.

The class is structured around this post-apocalyptic scenario where an entrepreneur in a nuclear wasteland has rediscovered coffee beans and engineered his potions to give people radiation resistance. His small shop is expanding to multiple locations with district offices and a headquarters. And they have hired *you* to design a production-level network comparable to real-world business – right down to the legal notice that unauthorized access equipment will “end badly.” It’s hokey, but it gives the mind a story to latch on to while absorbing otherwise dry examples.

I made a virtual machine (VM) to use for the class. Lesson 0 was about obtaining and installing Packet Tracer, a program from Cisco included with their free courses. Pick one, sign up, then ditch. Then I heard about the labs. See, Packet Tracer is a “simulator.” It’s great for anyone of any skill level to explore how a pile of network gear might work in the real world when you hook it up, but it isn’t 1:1. It has bugs, it is focused exclusively on Cisco, and the early 00’s are calling about getting their interface back. The hidden dark mode is 20% more bearable.

But in practice, the VM has mostly been used for watching class live streams. Turns out the class includes these labs in the course work where we have access to the actual Cisco IOS licensed by the class (Turns out Apple stole the name iOS and they settled with Cisco out of court). Feedback from the summer session has been glowing from the labs, so they’re clearly important.

I’ve spent the last few years poking around with networking goodies off and on, so until we hit the labs, I knew most of the theory we were going over. But yesterday (as of posting), I slammed into a lab quite hard. ~3.5 hours minimum, not counting the time already sunk on Friday. 45 minutes (resettable) on the clock. Pain. Also not helping was how a few concepts were used that hadn’t been formally introduced yet – or the legacy shortcut keys I haven’t been formally introduced to.

Takeaway

Trials and failure are a part of learning. Right now, I’m feeling defeated, but on some level, I understand that that only becomes true when I put it down forever, which I have no intention of doing at present.

Prayer

Father in Heaven,

This hasn’t been a fun month for projects. I haven’t completed anything novel to report on, so I am presenting what I do have. And at that, I choose to keep one work-in-progress I’m stuck on to myself.

I ask for your guidance and encouragement in these projects, just as I ask you bless the hands of my readers where they might be lost and without a guide. I ask that you be with my classmates – may the various, alien-feeling concepts and commands work towards as a second language to us, because I don’t know how many more labs I can take like this last one!

In Jesus’ name I pray,
Amen

Final Question

I’ll open it up. Do you use LLDAP as an authentication server? For QNAP? How did you get it to work?

Posted on

Is LLDAP Right for You? (QNAP Network Accounts)

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am working on my homelab. I have a lot I want to get done, and I need to stay on task, so I’ll be writing as I go. Let’s get started!

My goal this month –and possibly future months if this format works out– is to have a set of four related mini-projects. If you’re just here for my verdict, check the Takeaway. If you want the journey, hang on.

Overall Vision

This month is all about the family’s GoldenOakLibry access. Joining me is my Hermes AI agent, Shimmering Dust. He needs lots of hand holding. At least I’ve gotten his Gemma 4 26b “brain” doing its thinking on my graphics card, which helped with speed. He is my “rubber duck” with an FAQ list and search engine skills. A notable change I made for him this month was installing Lynx, a terminal Internet browser (his Firefox install was getting guff).

With my primary objective in mind, I need an LDAP server. Next, I’ll use that server to authenticate NFS (Network File System) access and/or web consoles. An eMail server is also a high priority. NextCloud would be nice, and should be within easy reach assuming GoldenOakLibry goes smoothly.

Project 1: LLDAP Container

The good news is that I already have done my reading on LDAP (Lightweight Directory Access Protocol) and turned up “Lightweight LDAP” (LLDAP). The bad news for me is that LLDAP prefers PostgreSQL and my early exploration landed MariaDB as my initial favorite. Long story short: each database organizes data differently. Not all projects will support both. It’s not worth my time to milk storage space savings by using fewer [layers of] a container.

I continued with Shimmering Dust while configuring LLDAP. By the end, lldap.container sat upon lldap-postgresql.container, which used lldap-postgresql.volume, with the containers subscribing to lldap.pod. Shimmering Dust was very useful for pointing out my typos and getting me un-stuck when I might have spent a few hours staring at old forums. Even now, I want to spell it “Postgress” (it has one ‘s’).

The last thing of note here is that Shimmering Dust tripped over a Pod= statement. I get it, Podman+Quadlets+Pod is pretty niche and new, so I can understand if Gemma 4 doesn’t have it in its training data. Nevertheless, Shimmering Dust made a self-improvement skill when we were done.

But hold up, is using a pod here even best practice? I wouldn’t call myself a hacker, but my guess is that a hypothetical attacker would have a harder time moving between containers linked by internal networks than by a pod that binds them tighter than normal containers. After all that work, I de-podded them.

Project 2: Network Accounts

My LDAP foundation took a week. On to using it. First, I need to expose LLDAP. GoldenOakLibry’s Domain Security page under ControlPanel listed a field for LDAP security. Looks like the preferred is the ldap+TLS option [1].

But what about the other options? I quickly got lost in the Great Forest of Knowledge TLS, SSL, and StartTLS. Shimmering Dust kept hallucinating between LDAP+TLS and LDAP+SSL with all the Confidence™ of 30+ years of sloppy industry using the terms interchangeably. DuckDuckGo’s AI bot said one was depreciated (EDIT: early versions are retired [2]. I strongly recommend you read this citation, it cleared up two days of painful confusion).

I found references to a Caddy+LDAP plugin – an unofficial community plugin I wouldn’t know how to install. OPNsense too has LDAP settings to explore. Further digging showed they require a separate LDAP server, so my time setting up LLDAP wasn’t time wasted. On a further side track, I learned about .home.arpa, a “special use” domain I should have been using all along. Disheartening.

Thinking long-term, I remembered I need to make a private website for my largely technologically illiterate extended family. Self-hosting Minecraft was also on my mind for my “online family.” Researching the latter, I found u/cfarence’s relevant thoughts on Reddit:

I have a custom portal where players login and tell it to whitelist their IP address. This then feeds into a pfsense dynamic list to allow the traffic through the perimeter firewall.

Works fairly well and it’s semi easy for players to login. This allows me to have it “open” to the internet and not have it hammered from all over the internet. It doesn’t handle dynamic IPs but players public IPs don’t change too regularly most of the time. [3]

Other Redditors do everything from hosting in the clear to using Tailscale or a virtual private server. Caddy was only built to handle HTTP[S] traffic. I can serve a login page, then manage access to homelab services from there. If someone needs around Caddy, he/she can use an IP authentication tool. Services can handle their own traffic.

Enough musing. Joystick (server’s hostname) needs a hole in its firewall for LDAP traffic.

sudo firewall-cmd --zone=public --add-forward-port=port=389:proto=tcp:toport=38900 --permanent

This rule redirects LDAP and LDAP+TLS traffic to a non-privileged port at 100x the service’s normal port number. I stashed this rule in my notes on in case of homelab rebuild. Interestingly, LLDAP did something similar by multiplying its privileged ports by 10x – the first container image provider I’ve seen do this. Thus, LLDAP’s Quadlet got the following line:

PublishPort=38900:3890

I asked Shimmering Dust for a good way to test the port, and he suggested nc (NetCat), utility I hadn’t heard of.

nc -zv <Joystick’s IP> 389

This tool came in handy as I worked – first on the pod, then through the firewall, then from GoldenOakL– QNAP went and ignored this utility, having “Notification Center” in its place. This wasn’t my first scrape with proprietary systems built on Linux missing expected programs, so I wasn’t too surprised. Shimmering Dust suggested a suite of tools from Telnet to Curl, and I found Sudo Ping for good measure (Why make ping admin only?). Between the tools, we determined GoldenOakLibry was seeing LLDAP.

With over half the week gone, I made the executive decision to ax TLS on LDAP, but with a plan for Caddy to manage a certificate for it using a shared volume. Oh, and after it was too late to change anything, it appears GoldenOakLibry has an LDAP server built in. Oh, well.

Next wall. Login. The top FAQ item is dedicated to it [4]. Turns out I’d installed incorrectly. I didn’t document the full process of “sideways progress,” but it involved adding a bunch of echo statements to bootstrap.sh to chase down a weird error:

"parse error: Invalid numeric literal at line 1, column 16"

I traced this down to the auth() function, where I found an error from a failed database login being processed as JSON:

`Authentication error for user "customadmin"`

16 characters in, that’s the first space, if you include the ` mark.

And here I stalled for the longest time. This month hit the family with two major emergencies back-to-back, and neither feel fully real yet (early morning of 8-21). I will be lucky to finish my base goals. Nevertheless, after a good session with Shimmering Dust where I had him go source diving with a copy of logs for both LLDAP and its Postgress containers, he noticed LLDAP sending MySQL commands to Postgres (I don’t have the skills with databases to verify), which in-turn produced the failed login and crash above.

Shimmering Dust recommended I try using an all-caps version of the variable LLDAP_Database_url to be consistent with relevant standards. As it turns out, the LLDAP container as it is expects the mixed-case version, and I happened to have made it all-caps already. This change allowed me to initialize my LLDAP database and log in.

Project 3: Connecting GoldenOakLibry

With a week left on the clock, I began work on logging GoldenOakLibry into LLDAP. My progress was entirely sideways, bouncing between running process of elimination, circling over similar ideas with Shimmering Dust, and reaching out to the empty LLDAP Discord community.

One big burrier I busted was authentication. I installed Wireguard-CLI to examine the unencrypted traffic between GoldenOakLibry and LLDAP, but its utility, Tshark, resists running as root (I’m monitoring LDAP traffic on a privileged port, after all). I soon found TCPDump, a small packet sniffer program, and ran the log through Tshark.

$ sudo tcpdump -i any 'port 389' -w ~/packet_log.pcap #Ctrl+C to end log
$ tshark -r ~/packet_log.pcap | grep LDAP

I found LLDAP rejecting GoldenOakLibry’s LDAP bindRequests despite following the examples.

Rejected: "id=user_name,dc=yourdomain,dc=lan" simple
Expected: "uid=user_name,ou=people,dc=yourdomain,dc=lan" simple

The next wall was authentication, and took the longest. Shimmering Dust kept volunteering scripts to test hashing algorithms, but I vetoed the attempts. I changed my admin password to “changeme”, and suddenly GoldenOakLibry made it in. My standard procedure for passwords is 128 mixed-case alphanumeric with special characters. Testing showed the max character length is 64.

But the last hurdle stopped me cold the moment I understood it. Even though Tshark was reporting successful bindRequests and LLDAP was sending over all its user data, domain users were still not populating in. I grabbed a large, suspect log from LLDAP’s container where tens of fields were being ignored and fed it to Shimmering Dust – who advised that QTS (GoldenOakLibry’s operating system, made by QNAP) may be expecting a full POSIX environment (Portable Operating System Interface), which LLDAP was never going to provide in the first place.

As of posting, I have a support ticket out to QNAP. So far, my experience so far on this issue has been disappointing, with support only ever getting back to me once a weekday a few minutes after 5:00 PM. On first reply, I spent around three hours drafting an eMail, then gutting the fluff when it was too long for their support form. Then it had to be submitted as a second issue when their system glitched, and that cost me a full interaction when the technician helping me got confused at my merged ticket – during which I solved my authentication hangup and hit the presumed POSIX problem. I was asked to provide a log, but by then I had already decided to switch gears to QTS’ internal LDAP server as a workaround.

Takeaway

I seem to recall trying for a similar format where I compress several smaller projects into one. They have a tendency to go wrong and get delayed. I’m 99% sure LLDAP isn’t compatible with QTS’ LDAP client. Check my next post or two for a solid yes/no.

Prayer

Father in Heaven,

It’s been a long month for me. I’m feeling immense pressure to finish this project on a time frame I can’t do anything about, and having all my progress pulled about at the last few days isn’t doing any favors for moral.

But Father, I ask that when I or my readers get to feeling like this, you help us recontextualize it as learning – be it as “sideways progress” as I like to call it or whatever makes sense.

I praise you for professional tech support. While I may not have a final answer right now, I have faith that I’m being taken care of by someone whom I can’t see, yet knows what he is doing. I ask that you bless him and his team with wisdom as they study my setup.

Likewise, I pray that my readers find inspiration with their own projects – that they absorb the concepts they are studying.

In Jesus’ name I pray
Amen

Works Cited

[1] QNAP, “Connecting a QNAP NAS to an LDAP Directory,”qnap.com,Aug. 7, 2020 (Ambiguously: July 8, 2020). [Online]. Available: https://www.qnap.com/en/how-to/tutorial/article/connecting-a-qnap-nas-to-an-ldap-directory. [Accessed Aug. 9, 2026].

[2] M. Normann, “Overview of TLS and LDAPS,” kb.symas.com, Aug. 19th, 2024. [Online]. Available:https://kb.symas.com/configuration/overview-of-tls-and-ldaps. [Accessed Aug. 11, 2026].

[3] SparkyGears, cfarence, et. all, “Safely Self-Hosting a Minecraft server,” reddit.com, Nov. 24, 2023. [Online]. Available: https://www.reddit.com/r/selfhosted/comments/182h7x9/safely_selfhosting_a_minecraft_server/. [Accessed Aug. 10, 2026].

[4] selfhoster1312, “[LLDAP] Frequently Asked Questions#I can’t log in!,” github.com,July 6, 2025. [Online]. Available: https://github.com/lldap/lldap/blob/main/docs/faq.md#i-cant-log-in. [Accessed Aug. 14, 2026]

Posted on

Homelab Check In 2026

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am reviewing the important parts of my homelab, going over the servers and software, and improving the inner workings. Let’s get started!

Hardware

The Internet comes in and connects to my ISP’s box. It’s set up in some kind of bridge mode. I have an Opnsense hardware firewall named Cerberus serving as the network hardware firewall and router. Wi-Fi is handled by an older router with stock firmware.

I have a server closet with my old RedLaptop, a NAS (Network Attached Storage) called GoldenOakLibry, and a pair of Dell Optiplex towers named ButtonMash and Joystick.

Golden Oak Libry

When we first got a NAS, we went with Synology. Since then, I’ve learned and the Linux kernel has developed, and one day I heard from tech support that they won’t be supporting a mission critical feature given my software stack (more on that later). I’ve been taking it slowly, but I’m transitioning into a newer QNAP device and adjusting to its quirks.

The driving force behind this month’s efforts has been completing this migration. I’d previously moved all the data onto other drives and back while only missing a few small files – mainly thanks to a terminal program called rsync. I grabbed all the metadata along with it – including UID/GID (User ID/Group ID) of all the files. And since I’d set up the previous NAS to not care about users, it was all Synology’s default “everybody” identifiers.

My goal for this time around is to distinguish between users. How-hard-can-that-be? Very.

Researching for Network Login

NFS (Network File System), the protocol I’m using to talk with Golden Oak, by default just trusts the incoming UID of any clients connecting to it. That means that if I upload a file from DerpyChips with a UID of 1000 and I access the NAS from my new laptop with a UID of 1000, everything will just work. But nothing is keeping my father’s computer from reporting the same UID, as 1000 seems to be the default value for all initial users. Now, I have an account on my father’s computer, and since it was created after his, my UID on there isn’t 1000. Too much work, too fragile. Approach rejected.

A more robust approach to solving this problem is Kerberos. I have some recollection from my initial days setting up the Synology NAS. In short: a client talks to a trusted server to get at a service. I spent a while researching implementation –including using Buildah to make a container to run with Podman– but found it expects way more network infrastructure than I already have, so I pivoted toward learning about this technology I keep hearing about in relation to homelabs: LDAP.

LDAP (Lightweight Directory Access Protocol) is roughly equivalent to Active Directory in Microsoft Land. And in true Linux fashion, LDAP is a specification, not a single piece of software. Opinions differ as to which is best for a use case like mine, but LLDAP (Lightweight LDAP) seems to be right for me.

Joystick

Podman is the underlying technology I use for my homelab. It is often compared to Docker, but it works without root access in case of container breakout. I have a non-admin account on my server, Joystick, where I use Quadlet files for permanent Podman containers so they’re always running when I expect them to be.

But the heart of my homelab (software wise) is Caddy. Caddy is more or less a do-it-all networking suite. It’s a reverse proxy with built-in HTTPS – both real and by self-signed SSL certificates. And everything Caddy does is controlled by a Caddyfile written in as simple a format as possible. Once everything is in motion, the most annoying part is telling your computers and browsers to trust your self-signed certificate authorities.

Taking inventory, I found I didn’t have much in fully functional order. Vaultwarden (3rd party Bitwarden server written in Rust) and Crafty Controller (Minecraft server management software). Beyond that, I have an incomplete Nextcloud setup waiting on Golden Oak and dreams of many more projects – many of which also are waiting on Golden Oak Libry. The home directory was getting full, so I sorted them between Services that provide target functionality and Network that support the Services.

One of Caddy’s features is a static web server. After reviewing all my containers, I wrote a landing page for it to display. Debugging it ate a bunch of time, but I found I had goofed a mount configuration. From there, I linked my different services as a way to test them.

During this process, I discovered a poor Caddyfile entry for Crafty. Someone on GitLab had suggested essentially bypassing HTTPS security in a way that otherwise looked legit. This would have been fine for my setup, but out of concern for the community, I updated an earlier solution that relied on a deprecated Caddy term a different Caddy /path/to/certificate. In this way, I learned one way to “establish trust.”

Cerberus

Hardware Firewall. It’s basically a fancy router – a REALLY fancy router. If Caddy is the heart, Cerberus is the head. It handles the DNS blocklist using the OPNsense plugin Unbound. But Unbound also handles domain name resolution. I was several major versions out of date. Oops. Fixed.

Ironing out Unbound was another place that ate up diagnostic time like crazy. I eventually happened across a bug report where DNS server crashes if Unbound tries forwarding a wildcard subdomain (as in *.myDomain.lan) if the router thought itself on that same domain (under System>Settings>General>Domain). It wouldn’t stop completely, but it would get really slow. Switching to myDomain.arpa is a small price to pay for not having to manually forward each subdomain I configure for Caddy.

Takeaway

Life’s been hectic as of late, so I haven’t had the time or usable attention span to finish all my goals. Instead, I found several smaller goals and am in the process of whittling away toward the larger ones. My homelab was a bit of a mess after the techno-wizzarding I’ve been pulling off, and on this later pass, I’ve improved my understanding of what it was I was doing and how to engineer it better.

I have plenty of smaller goals I left out because I want this to be a log of what I’ve managed to finish – not a dream log of what I still have to do, but I still have one thing I wish to share.

One of the links on my homelab landing page is back to the router: https://cerberus.myDomain.lan. Firefox won’t load it. I looked up how to establish trust between Caddy and OPNsense, and my search results were flooded with a Caddy plugin for OPNsense. I’m sure it’s a fine project, but I don’t want to invest in that toy right now. Long story short, I pivoted approaches to use an ACME (Automatic Certificate Management Environment) client I found for OPNsense. I left an entry in my Caddyfile to hopefully expose the ACME server and did everything to make my first attempt. It failed, but at least I got to that attempt in the first place. I do however need to call it quits on the research.

Prayer

One YouTuber I occasionally pay attention to is Network Chuck (Shoutout if you’re reading this). I may disagree with his claim that “everything in IT requires coffee,” but I do like how he’s made a space in his videos for Jesus, and I’d like to do the same (the prayer, not the coffee).

Father in Heaven,

I thank you for the progress I’ve had this month, and I praise you for whomever is sticking around. Please bless him or her according to your will. May you inspire my readers in their personal projects, be it with Caddy, OPNsense, or more. Guide them towards whatever documentation or bug report they need, and help them avoid focusing on premature feature creep.

In Jesus’ name I pray, Amen.

And if you are curious about the Bible, God, or Jesus, I’d be more than happy to talk. These days, I primarily use Discord. There’s a link to my server in my Socials tab at the top of this page. So far, He’s been welcome there, but from now on, I’d say He’s properly invited.

Final Question

Do you have an opinion on which LDAP server reigns supreme for homelabs? I look forward to hearing about it in the comments below or in my Socials!

Posted on

A Pair of Programming Projects

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am covering my programming work this past month. Let’s get started!

I am finding that even on the monthly cycle, I don’t always finish the projects I intend to. This month, I was intending to work on my phone, but that project hit some tough truth and the story isn’t ready yet.

HyperTrace

I might as well introduce my newest long-term project, HyperTrace. It started out in October as a spectator camera for a larger Redot community project. I studied up on using quaternions to rotate a camera node to avoid gimble lock, but just as I was starting to get it under control after I was bored stiff of them, I realized I could just attach a camera to a physics object.

But once I had a camera with 6 degrees of freedom (three positional and three rotational), I started thinking about how I could make a game out of it. I figured a space piloting game was in order. I got into studying up on using Perlin noise and similar methods of making structured noise to generate a shifting cave structure to represent a hyperspace mesh for the player to navigate – preferably at high speed.

The idea is that an unnamed pilot is tracing out a route through a shifting hyperspace (preferably at high speed), and if he collides with a wall, he is thrown out of hyperspace, where it isn’t safe to re-enter, so he must return to port under conventional physics. If he succeeds, his fleet can instantly jump across his route. This FTL system sets the stage for a future where humanity has colonized the solar system, but is hesitant to venture out to other stars. Think Star Wars meets Space Engineers meets Monkey Ball.

So far, my best tech demo is a blue fog you can navigate at slow, careful speeds. It takes an unacceptably long

Shell Scripting

My services were called upon for a “computer project.” My mother had some PDF files of recipes to separate. Over the course of an afternoon, I wrote a BASH script to separate out each page in each PDF. At its heart was the program qpdf. Along the way, I tried to keep good practices like leaving comments and having checks to prevent accidental overwriting.

Now, I only know a modest bit of BASH (popular Linux command line interface aimed at users), but I know enough about basic programming it was just a matter of syntax for me. But I figured I could do one better in around the same amount of time. I saved a copy of the script and its output, and I set about naming files after their contents, identifying+recombining second pages, and sorting out charts, and numbering repeats (some duplicate recipes had changes).

But for every hour I spent working out a feature, I figured I spent another debugging edge cases. Several recipes had BASH control characters in their titles like *, &, and /. I preferred to use special handling of strings, but in the end I sanitized / as ÷ and * was removed entirely. This additional work extended the project to a total of three days.

Closing thoughts

My goal with Hypertrace is to publish it to Itch.io or similar platform on a donation basis. And before I get past more than a couple tech demos, I’m going to want a version control server in my homelab just in case something happens to my laptop. While I really want to think ahead in terms of security, I know I don’t know enough to make an informed decision about the utility I need now.

Final Question

What Git setup do you recommend for a homelab?

Posted on

My First Computer “Rack”

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

So far, I’ve been assembling my servers (ButtonMash, RedLaptop, and GoldenOakLibry) on and under a foldable table. Add a workstation, and it’s getting a bit cluttered. We’ve had a set of glass shelves going unused for a while now, and I think they might do nicely to organize the servers’ room.

I started by measuring ButtonMash’s case against the shelves’ metal frame. While it was close, I estimated an inch vertical clearance once the shelves were in place. Otherwise, the tentative plan was to remove a shelf. My father and I moved the shelves in and loaded them and the setup’s UPS (Uninterruptible Power Supply). Wiring was relatively straightforward with the traditional wire Medusa in the back, but out from underfoot.

Unfortunately, I left ButtonMash in a precarious state such that a reboot before moving it knocked out my known house of cards supporting PiHole and Unbound. What I didn’t realize was that I never got Caddy working on that machine in the first place. In trying to fix Caddy, I wiped the containers I actually the whole house was using for DNS. As a patch, I pointed the router back at our normal DNS servers.

While I’m trying to avoid server work this month, I went ahead and looked up how to change my specific DNS settings temporarily to restart my DNS containers. From there, I did not encounter any notable issues, though I wasn’t up to testing the removal of my patch.

Takeaway

I have a rack. That’s my story and I’m sticking to it.

Final Question

How do you organize your tech stack?

Posted on

Rocky Server Stack Deep Dive: 2023 Part 5

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am learning more about Podman Quadlets for my homelab. Let’s get started!

Systemd and Quadlets

From my incomplete research going into this topic, I already know Quadlets is a system for efficiently integrating Podman containers in with Systemd. It was merged into Podman v4.4, and I had a small pain of a time trying to find a distribution with both that and legacy BIOS support along with a list of other requirements.

But what is Systemd? In short: Systemd is the init process –a process that manages other processes– used by most Linux distributions that aren’t trying to optimize for a low RAM or storage footprint. As it turns out, I’ve already had minimal exposure to it while writing unit files for NFS [auto]mounts and a static IP address on Debian. Systemd in turn bases units off these unit files to manage the operating system.


While Systemd unit files defining Podman containers can be written by hand, Quadlets can automate their creation based off simpler unit files of its own: .container, .network, .volume, and .kube. The first three look similar enough to concepts I’m familiar enough with that I figure I could hack an example into doing what I need.

But I’m interested in pods. With .pod unit files only a controversial feature request at best, that leaves me to explore .kube files, which run Kubernetes YAML files. I know nothing about writing Kubernetes YAML files from scratch, and I refuse to cram for them Thanksgiving week.

My project died here for a few hours. One Systemd tutorial brought up Syncthing in an example, and I spent a while on a tangent looking at that, but it too is too large to cram for this week. I unenthusiastically browsed back to Kubernetes, and found:

podman generate kube

Looks like I just might get away with adapting my scripts after all this week. With this in mind, I copied over my files from my laptop’s Debian drive to its new-last-week Rocky 9 installation. Focusing on Nextcloud, I cleared out my dead-end work with Fuse, abstracted volumes, and other junk before realizing BusyBox was likely a more suitable testing grounds.

My First Kuberneties File

I came up with the following bash script for such a pod:

podman pod stop busyBoxPod
podman pod rm busyBoxPod
podman pod create busyBoxPod
podman create \
    --pod busyBoxPod \
    --name BusyBox \
    --volume fastvolume:/root/disk \
    -it \
    --rm \
    busybox

And here is

# Save the output of this file and use kubectl create -f to import
# it into Kubernetes.
#
# Created with podman-4.6.1
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2023-11-23T01:29:45Z"
  labels:
    app: busyBoxPod
  name: busyBoxPod
spec:
  containers:
  - image: docker.io/library/busybox:latest
    name: BusyBox
    stdin: true
    tty: true
    volumeMounts:
    - mountPath: /root/disk
      name: fastvolume-pvc
  volumes:
  - name: fastvolume-pvc
    persistentVolumeClaim:
      claimName: fastvolume

I saved this output as busyBoxPod.yml and returned to Nextcloud.

Nextcloud put up a small tantrum getting re-updated for Podman 4.6.1. I had to look up how to Podman Secrets, and apply :z to volumes to satisfy SELinux. Redis however, refused to accept a password from Podman Secrets, so I rolled back that change. The pod should insulate it anyway. I got it to a point where it needed a domain name.

Branching out to bring up Pi-Hole and Caddy, I learned how the default Unbound configuration for the container I used only forwards DNS requests to Cloudflare. I’ll want to fix this later. I used firewall-cmd to forward ports for HTTP, HTTPS, and DNS to underprivileged ports for rootless containers.

Takeaway

UNCLE! I find more and more of my time supposedly working on server is procrastinating and stressing over either minutia or blankly staring at my screens when I muster enough focus to ignore distractions. There’s no way around it; I’m officially burned out on this project. I’ll maybe come back to it after the new year. I really wanted to get my .kube files working for at least Pi-Hole and Caddy, but it’s going to be a hard pass at the moment.

Final Question

I’m considering covering a free/open source game or few over December. What are your recommendations?

I look forward to hearing from you on my Socials!

Posted on

Rocky Server Stack Deep Dive: 2023 Part 4

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am exploring fuse-overlayfs as a possible patch between Podman and NFS. Last week’s post was practically a freebee, but I expect this one to be a doozy if it’s even possible. Let’s get started!

Context

For my homelab, I want to run Nextcloud in a rootless Podman 3.0.1 container with storage volumes on our NFS. For logistical reasons, Nextcloud needs to be on RedLaptop running Debian 11 (Linux kernel 5.10.0-26-amd64 x86_64). The NFS share I wish to share is mounted via systemd.

My most promising lead is from Podman Github maintainer rhatdan on October 28, 2023, where he made a comment about “fuse file system,” asking his colleague, @giuseppe, for thoughts to which there has been no reply as of the afternoon of November 10 [1]. I documented a number of major milestones there, which I’ll be covering here.

File System Overlays

Fuse file system turned out to be fuse-overlayfs, one of a few systems for fusing file systems. Basically: there are times when it’s useful to view two or more file systems at once. File system overlays can designate a lower file system and an upper file system. Any changes (file creation, deletion, movement, etc.) in this combined file system manifest in the upper file system, leaving the lower file system[s] alone.

Through a lot of trial and error, I set up a lower directory, an upper directory, a work directory, and a mountpoint. My upper directory and work directory had to be on the NFS, but I ran into an error about setting times. I double checked that there were no major problems related to Daylight Savings Time ending, but wasn’t able to clear the error. I sent out some extra help requests, but got no replies (Sunday, Nov. 12). A third of my search results are in Chinese, and the others are either not applicable or locked behind a paywall. Unless something changes, I’m stuck.

Quadlets

Github user eriksjolund got back to me with another idea: quadlets [1]. Using this project merged into Podman 4.4 and above, he demonstrated a Nextcloud/MariaDB/Redis/Nginx setup that saves all files as the underprivileged user running the containers. In theory, this sidesteps the NFS incompatibilities I’ve been experiencing all together.

The first drawback from my perspective is that I need to re-define all my containers as systemd services, which is something I’ve admittedly been meaning to do anyway. A second is again that this is a feature merged into Podman much later than what I’m working with. Unless I care to go digging through the Podman GitHub myself, I’m stuck with old code people will be reluctant to support.

Distro Hunt

Why am I even using Debian still? What is its core purpose? Stability. Debian’s philosophy is to provide proven software with few or no surprises left and the user polishes it to taste. As my own sysadmin, I can afford a little downtime. I don’t need the stability of a distro supporting the most diverse Linux family tree. Besides, this isn’t the first time community support has suggested features in the future of my installation’s code base. Promising solutions end in broken links. RAM is becoming a concern. Apt package manager has proven more needy than I’d care to babysit. If I am to be honest with myself, it’s time to start sunsetting Debian on this system and find something more up-to-date for RedLaptop. I’ll keep it around for now just in case.

My first choice was Fedora to get to know the RedHat family better. Fedora 39 CoreOS looked perfect for its focus on containers, but it looks like it will require a week or two to configure and might not agree with installing non-containerized software. Fedora 39 Server was more feature complete, but didn’t load up for my BIOS (as opposed to the new standard of UEFI); I later learned that new BIOS-based installations were dropped on or around Fedora 37.

I carefully considered other distributions with the aid of pkgs.org. Debian/Ubuntu family repositories go up to 4.3. Alpine Linux lacks systemd. Souls Linux is for desktops. OpenSuse Tumbleweed comes with warnings about being prepared to compile kernel modules. Arch is… Arch.

Fresh Linux Installation

With time running out in the week, I decided to forgo sampling new distros and went with minimal Rocky 9. Installation went as best can be expected. I added/configured cockpit, podman, cockpit-podman, nfs-utils, and nano. I added a podmanuser account, set it up to allow-lingering, and downloaded the container images I plan on working with on this machine: PiHole, Unbound; Caddy; Nextcloud, Redis, MariaDB; busybox.

Takeaway

I write this section on Friday afternoon, and I doubt I have enough time remaining to properly learn Quadlets and rebuild my stack, so I’m going to cut it off here. From what I’ve gathered already, Quadlets mostly uses Systemd unit files, a format I’ve apparently worked with before, but also needs Kubernetes syntax to define pods. I don’t know a thing about using Kubernetes. If nothing else, perhaps this endeavor will prepare me for a larger project where larger scale container orchestration is needed.

Final Question

Do you know of a way I might have interfaced Podman 3 with NFS? Did I look in the wrong places for help (Debian forums, perhaps)?

I look forward to hearing from you on my Socials!

Work Cited

[1]. Shadow_8472, D. “rhdan” Walsh, E. Sjölund, “Rootless NFS Volume Permissions: What am I doing wrong with my Nextcloud/MaraiDB/Redis pod? #20519,” github.com, Oct. 27, 2023-Nov. 10, 2023. [Online]. Available: https://github.com/containers/podman/discussions/20519#discussioncomment-7410665. [Accessed Nov. 12, 2023].

Posted on

My Homelab Report, 2023

Good Morning from my Robotics Lab! This is Shadow_8472 with a snapshot of my different computers as of mid-March, 2023. Let’s get started!

I have felt a need for a page detailing my so-called “homelab” detailing the computers and other equipment I personally maintain. I didn’t finish any big projects this week, so here’s a first draft of that page. It’s listed in roughly the order I took ownership/responsibility for each piece. I’ve tried to gather each computer’s base model/modified configuration plus a list of its presently bootable operating systems. Warning: unexplained technobabble ahead.

Red Laptop

Dell Inspiron N7110
1 TB SSD

Debian 11

As the oldest operational computer in my fleet, this laptop has been with me for almost 12 years, though most every part was replaced while it was under extended warranty. Shortly after I started this blog, I installed Debian 10 on a USB external drive. I nuked its Windows drive one week by formatting the wrong disk, and later replaced it with a 1 TB SSD, cloned Debian inside, and it served as a get-me-online machine until the screen bezel cracked, threatening to shatter the screen the next time I’m not careful. It’s now serving as a backup home server, and it’s desktop environment has proven vital to rebounding from at least one network emergency since.

Upstairs Workstation

Custom Build

Intel Core i7-3770K (3.50GHz)
NVIDIA GeForce GTX 970 (rev a1)
24 GB RAM (2×8+2×4)

Windows 10
Manjaro
EndeavourOS (WIP)

“Upstairs Workstation” also predates when I started naming my computers. I got it for university when laptop turned up as bit underpowered. I went from Windows 7 to Windows 10 with this machine, narrowly avoiding an online tech support scam (10 didn’t support a Wi-Fi cart I was using at the time and later finished burning out when I put it back in after switching to Linux).

Derpy Chips

Custom Build
Intel Core i7-2600K (3.40GHz)
16 GB RAM

PopOS

Derpy was named for its tendency to give power kernel errors during its first service life until its retirement. When I adopted it, I gave it a new hard drive and inadverantly diagnosed the one lowballed part: its 2 TB HDD. I’ve also had issues with a stick of RAM from it, but I’ve had it running smoothly for a few years now as a secondary workstation.

ButtonMash

Dell Optiplex 7010
16 GB RAM
1×60 GB SSD
1×1 TB SSD (external)

Debian 11
Rocky 8

ButtonMash has been my main x86 experiment machine as well as my home server. I bought it from a church member who was upgrading his dental office computers. I My was host to my first Linux install: Ubuntu, but I’ve put it through a parade of distros including MicroCore (where I learned the command line), MineOS, Debian (originally from “red laptop”), and finally Rocky. Today, it serves as my primary home server, where I am setting up several Podman containers including Vaultwarden, PiHole, Nextcloud, and Minetest – each on a separate account for further isolation.

If and when we need to scan pictures, ButtonMash is dual booted with an external SSD I originally used with my Red Laptop.

GoldenOakLibry

Synology NAS

4x7TB HDD RAID 5 share
1 TB SSD share (external)

GoldenOakLibry is our home NAS intended for picture scanning. If nothing else, it’s exposed me to using a computer that’s meant to do nothing but serve files. I have a long way to go before I master it.

Raspberry Pi’s

1xRpi3B+
2xRpi4B
1xRpi400

My collection of super-flexible Raspberry Pi computers has served me well. At the moment, the Pi 4’s are serving as OpenWRT router firmware configured as Wi-Fi-to-Ethernet adapters for my upstairs workstation and my sister’s desktop. My Pi 3B+ has been inactive since its 3D printed PacMan ghost case got smashed, and my Pi 400 shares a monitor at my upstairs workstation.

Other/Misc computers

3D printer

Maker Select Plus
Added braces+feet

I am no master at 3D printing, but I know enough of the basics to scrape together most everything I try. At present, my printer needs a new fan and a printbed that isn’t warped.

Pine Phone

UBPorts Edition

The average smartphone these days spies on you. Its parent company is allowed to by the terms of service you need to accept before use. I resolved to not blindly accept any new EULA’s without knowing what I’m getting into and learning enough to be confident I’ve done something to mitigate the worst of these violations of privacy. As a consequence, I don’t have a usable phone ever since we switched providers a while back.

UPS’s

I have three uninterruptible power supplies directly under my supervision – one serving the network closet, one serving my upstairs workstation, and the other ButtonMash’s area. As of yet, I have not figured out the open source NUT driver.

My Father’s workstation

Dell Optiplex 7020
15 GB RAM

Linux Mint 20.3
Debian 11

15 GB RAM

After I bought ButtonMash, I got another used workstation out of the same dental office, replaced Windows, and gave it both sets of RAM. I later talked my father through installing Debian 11 on ButtonMash’s original hard disk

Ventoy USB (32 GB)

Not technically a computer, but easily one of my most versatile tools. Ventoy makes Linux media creation as simple as dragging and dropping a .ISO. Even better is that I can still put normal files on it!

Takeaway

Each part of my lab serves its function. There’s a certain joy when I get something new working and get to show it off.

Final Question

Which of your projects are you most proud of?

Privacy Policy