Tag: Podman

Posted on

Nextcloud Soon?

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am pressing on towards my very own Nextcloud deployment beyond a proof of concept. Let’s get started!

Preparing an Account

Last week, I learned that Rootless Podman comes with certain challenges when trying to access a network drive. ButtonMash’s Rocky 8 Linux hard drive is tiny, so I don’t want to use it for storage. That leaves my retired laptop, “RedLaptop” for the job.

I use dedicated user accounts to isolate processes. The first order of business was to create a new account and log in through the Cockpit web interface. I had a bit of trouble when the underlying SSH couldn’t connect because Debian doesn’t like capital letters in its usernames, but the nonexistent, capital-containing username was flushed out while diagnosing user groups per advice on Discord. The account was additionally locked to prevent normal logins via password.

Preparing Containers

I’ve been developing a couple scripts for running Nextcloud on ButtonMash in a pod, so I copied them and the secrets directory (password storage) over to RedLaptop and tried them out. Errors galore! I downloaded the Nextcloud and MariaDB container images, but the older version of Podman (3.0.1 on Debian vs. 4.4.1 on Rocky 8) meant missing features – namely Podman secrets and mounting volumes on pod creation.

The secrets were easy to revert. I just had to remove the –secret flags and reinstate the passwords in surrounded by quotation marks. Unlike best practices in later versions, volumes mounted nicely into their respective containers. I double checked one with

podman inspect <container ID> | grep -i volume

and found references to the correct volume.

New Territory

At long last, I made it to the Nextcloud installation screen where I need to make an admin account. From here on, all I know comes from the little scouting I did way back by using an SQLite database hosted offered from the Nextcloud container as opposed to a more capable one.

I got an error about failing to connect to the database. I found a username with a capitalization error and I didn’t actually have a named database created within MariaDB’s container. But even with both of those fixed, I kept getting kicked back to Nextcloud’s create-an-admin screen.

And here my research for the week runs out of time. MariaDB’s log shows two errors I suspect to be the root issue:

<timestamp> 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1
<timestamp> 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF

Takeaway

I so want to be done with this project. That is all.

Final Question

I am left without a solution for now. For all I know, it may be low RAM, which I’m already maxed out on for my model of laptop – meaning I’d need to start over again with something else. If anyone has any ideas, I’d be happy to hear about them in the comments below or on my Socials.

Posted on

Podman-Nextcloud: Climb Shorter Mountains

Good Morning from my Robotics Lab! This is Shadow_8472 and today it’s bully or be bullied as I take another swing at getting my Nextcloud instance even partially usable. Let’s get started!

If there’s one long term project of mine that just loves humiliating me, it’s getting Nextcloud operational. My eventual goal is to have it running in a rootless Podman container with a way to quickly move it to an auxiliary server. My strategy thus far has been to prepare three Podman volumes hosted on GoldenOakLibry (NAS) over NFS while accounting for the speed needs of the MariaDB and Nextcloud volumes with an SSD and vs the capacity needs of the PhotoTrunk volume with a RAID 5 array of HDD’s.

NAS: Network Attached Storage

NFS: Network File System

SSD: Solid State Drive

HDD: Hard Disk Drive

Lowering Expectations

I’ve lost count of how many times NFS has given me grief on this project, so I eliminated it. I moved the SSD from where it was on GoldenOakLibry to ButtonMash, my main server computer. I added it to /etc/fstab – bricking Rocky Linux. ButtonMash is dual booted, so I booted to Debian for repairs.

Rocky’s installation system uses an LVM2 format, which Debian can’t read by default. An LVM2 package exists, and I installed it. LVM2 partitions show up in lsblk as sub-partitions of an actual partition, and it is these sub-partitions that get mounted, for example:

sudo mount /dev/rl_buttonmash/root /mnt/temp

to mount the sub-partition that shows up as rl_buttonmash-root. While I did explore for a quick fix, it’s a very good thing when each side of a dual booted machine can repair the other. Mounting a file system is a very important tool in that kit.

Upon closer inspection, a contributing factor to bricking Rocky was the root account being locked. The computer booted into an emergency mode and got stuck in a loop ending with “Press ENTER to continue…” Unlocking it didn’t get me anywhere when I looked at the logs per the loop’s recommendation, but the command lsblk -f clued me in that I was mounting the drive using the wrong file system type, an error which was soon remedied after I discovered it.

Project Impossible

The move hardly seemed to fix anything as hoped. I didn’t solve much. I kept getting NFS related errors when trying to run the pod, even after moving to a new mountpoint I’d never touched with NFS automounts. I even tried mounting the volumes using hard links pointed at the mounted “data drive” and I still couldn’t get a working Nextcloud instance. Somewhere in my shambling among the apparently limited content available regarding this topic online, I found the following warning on Oracle’s Podman documentation:

Caution: When containers are run by users without root permissions, Podman lacks the necessary permissions to access network shares and mounted volumes. If you intend to run containers as a standard user, only configure directory locations on local file systems [1].

Rootless Podman lacks network share permissions. OK, so NFS out unless I can selectively give Podman network permission without going full root. Until then, Podman is limited to the local disk, and if I’m understanding this warning correctly, mounted drives are also off the table. My plans for a Photo Trunk upgrade may be grounded indefinitely, and with ButtonMash’s Rocky drive being only 60GB, I’m not looking to burden it with anything resembling bulk storage.

Takeaway

The next logical innovation would be to rebuild the project on a computer with more storage. Barring a full makeover of ButtonMash, I do have my Red Laptop as an auxiliary server. I made a new account, but in all reality, this inspiration came after my research cutoff. It’s a project for another week once again.

Final Question

My project directory is messy with scripts to the point where I started a README file. Have you ever had a project so involved as to need one?

I look forward to hearing about it in the comments below or on my Socials.

Work Cited

[1] Oracle, “Configuring Storage for Podman,” oracle.com, [Online]. Available: https://docs.oracle.com/en/operating-systems/oracle-linux/podman/podman-ConfiguringStorageforPodman.html. [Accessed July 27, 2023].

Posted on

Nextcloud Dangling in the Works

Good Morning from my Robotics Lab! This is Shadow_8472, with a stubborn-project update. Let’s get started!

Nextcloud as a project comes with a lot of strings and hooks. Every time I sit down to work on it, the feature creep closes in: multiple volumes for speed and capacity, MariaDB container so SQLite doesn’t choke serving clients, a pod for MariaDB to live in alongside Nextcloud, Redis for RAM caching (Out of my scope, but worth a mention), Ansible Vault for password management (automation tool I’d need a month to learn), podman secrets, Philomena image board software – not to mention the volumes I learned about the other week! As noted in previously, the container I’m using from docker.io can take care of itself, but it will work a lot better with even a few of these things.

In my ideal configuration, all my containers’ persistent data lives on GoldenOakLibry so my old, red laptop can take over on critical Podman containers when ButtonMash is booted to Debian for photo scanning duty. The big, hanging question then is if GoldenOakLibry’s internal HDD’s absolutely need to spin up to serve shares hosted on NAS volumes mounted from an external SSD.

HDD: Hard Disk Drive
NAS: Network Attached Storage
SSD: Solid State Drive

My initial tests involved verifying that a drive could be mounted externally, then rebooting the NAS. I was expecting either a 1 or a 50 second wait, but instead I kept hitting “Stale file handle” errors when I tried mounting the external share. My second round of trials a week or two later exhausted search pages worth of results to learn how NFS shares change a few invisible details when restarted. Either something wasn’t ready when needed during boot, or my client just needs a while to refresh those details. It works in the long run, and external shares are not dependent on internal disk spinup. Time to move on.

Having solved the last known problem, I started layering the bits I had solved while frustrated with the USB share. I prepared three Podman volumes: two for Nextcloud prioritizing high-capacity and speed, respectively, and one for an SQL database. A pod housing MariaDB (the SQL database) and Nextcloud containers mounts these volumes (!). Podman secrets safely injects the database password so MariaDB and Nextcloud can work together – not strictly necessary for my application, but a good habit to get in in case I go in the direction of creating images from existing containers. Once the cloud is running, I want to try adding an image board to view the pictures.

(!) NFS –as it turns out– does not appreciate a number of the acrobatics rootless Podman performs with Namespaces. I don’t have the specifics memorized, but when Podman makes a container, the host’s user ID is mapped to root within that container. Podman then assigns variations of the host’s user ID to non-root users within the container as defined by a namespace. The NFS protocol wasn’t designed with namespaces in mind. To NFS, it looks like you’re trying to access another user’s files – possibly without permission. While this reportedly doesn’t stop rootless Podman from working well over NFS over normal circumstances, there are a bunch of search results talking about the snag when working with container images over NFS. Based off my experiences for this post, the same appears to hold true for volumes.

Takeaway

As much as I wanted this to be the last installment, this issue is a post in and of itself, and I’m getting burned out again over trying to come back to it. I’ll try coming back to this in around a month or so.

Final Question

Have you successfully used Podman volumes over NFS? Please, do tell me all about it in the comments below or on my Socials.

Posted on

I Studied Podman Volumes

Good Morning from my Robotics Lab! This is Shadow_8472 with a side project of the week. Let’s get started!

Nextcloud has been a wish list item since I gave up using Google’s ecosystem (Drive, Calendar, Office, etc.). This open source almost-drag-and-drop alternative proved above my skill level at first, but I’ve learned a lot about server management and running OCI “Docker” containers in Podman in the years since.

Demo of Nextcloud

Nextcloud was relatively simple to demo: one non-privileged port forwarded. During my self-guided tour, I was amazed at the potential power there. In addition to its calendar, office, and file storage functions I expected, its recommended suite apps include email, chat, and contacts servers – with more apps available for download.

As much as I can see myself moving in right now, it’s important that I master how its persistent data is stored. Ideally, everything would live on GoldenOakLibry, my home network storage configured with RAID 5. But I hate waiting for HDD spin-up. If it had an SSD out the back (it has two USB ports), I could mount a directory there from a Nextcloud container and back it up to deep storage on a weekly or monthly basis. At the same time, I may want the capacity of the main disks if Nextcloud turns out to be good for hosting the family’s photo archive.

The solution: use two “volume” structures like I started looking at last week. They work similarly to directly mounting directories as I’ve been doing in theory, but are abstracted similar to containers. See my Tangent heading later on for more information.

With this purpose in mind, we ordered a USB–SATA adapter and dug up our MineOS SSD from once upon a time. I archived around 411 GB worth of Minecraft worlds to free up space. I got the poor idea to try compressing it to both .tar.gz and .zip, two widely used compression formats which turn out to be implementations of the same algorithm. GoldenOakLibry had ZIP, but not TAR, so I tarballed/compressed it to 393.5 GB over an hour vs the NAS struggled to produce a 393.77 GB .zip over a full day. With a savings that small, I’m probably best off curating it uncompressed – especially if I have family members interested in seeing it again.

RAID 5: A hard drive redundancy scheme resistant to a single drive failure. In my case: four matching disks with three drives’ worth of usable space.

TAR: Tape ARchive: An early archive tool often paired with the GNU Zip compression program.

To Do List

Regretfully, I have to split this topic just as it’s getting good. I was running a test to see if GoldenOakLibry can respond on the USB share without spinning up, but creating new network shares is not a skill I can perform reliably yet. I had it working once, rebooted, and now it won’t re-connect like the proven ones. If for whatever reason I can’t get GoldenOakLibry to share from MineOS’s SSD without spinning up, I’ll have to mount it internally to ButtonMash and play the BIOS game to disable booting to it.

Even if I was making good enough time with the USB share, I’d still need to study up on databases. The lightweight one included with the Nextcloud container I’m working with is meant for browser traffic only, and I want to try out its clients.

Tangent

I started with the working theory that volumes needed to be started and stopped like containers, and I would therefore need one of Podman’s signature pods to manage them alongside Nextcloud’s. It’s basically a container for organizing and running containers on a homelab scale. I rigged up a script to automate my attempts with it, but eventually realized that volumes are passive, and I had just learned the wrong tool for the job. I’m sure I’ll make use of it eventually.

Final Question

I’d like to open up the discussion. Two big choices remain: Where do I host my fast Nextcloud SSD (GoldenOakLibry or ButtonMash), and how do I host my archive pictures (Nextcloud, Mediawiki, something else)?

I look forward hearing your answers in the comments below or on my Socials.

Posted on

My Podman Containers Boot With Systemd

Good Morning from my Robotics Lab! This is Shadow_8472 and today I am reasonably sure my Podman containers won’t be randomly going down anymore. Let’s get started!

I enjoy using Podman as a stand-in for Docker, but its rootless approach to running containers inherently challenges sysadmins facing Docker’s help and tutorial legacy. The most problematic difference I’ve experienced has been keeping containers running long-term. Months ago, I learned how to enable account lingering. This allows Podman containers without something remaining logged in as their respective users. I’ve been living with manually restarting containers as needed. Well, since I decided to enable automatic security updates, starting containers automatically would be prudent before expecting other family members to rely on them.

Against all odds, my initial search this past Wednesday yielded a blog article from Red Hat about integrating Podman containers into Systemd [1] to start them at boot. It was posted the day before.

Podman and Systemd

I trust Red Hat to not post malicious commands, but it’s still a good idea to learn about strange commands before running them. Red Hat’s tutorial starts with making a new user, enabling linger, and running a containerized web server. The first important command I ran was

$ podman stop httpd && podman rm -a && podman volume prune

This command appears to thoroughly clean out Podman. I’ve mounted volumes from the host before to persist data, but there’s a more flexible volume structure I only learned about just now when researching for another section I had to spin off into a near-future post. I haven’t used them yet, but I’m sure they’ll be useful once I learn how to use them.

$ podman generate systemd --new --files --name httpd

This command makes a new systemd file. The –new option recreates the container fresh each time it’s brought online. –files sends the configuration to a file instead of the terminal. –name must be the name of a running container or pod.

$ cp -Z container-httpd.service ~/.config/systemd/user/

The file generated previously goes in a directory where systemd will find it when used with the –user flag. The -Z flag matches permissions with the destination directory. The tutorial finishes with a daemon-reload followed by starting and enabling the user’s service.

Takeaway

This is a resource for my bookmarks. That is all.

Final Question

I took the opportunity during this project to put a Minetest server on ButtonMash, but I’m having difficulty obtaining permissions. I can see its logs in Cockpit-Podman, but I don’t have access to the server command line. How am I supposed to get started with adminning Minetest?

I look forward hearing your answers in the comments below or on my Socials.

Work Cited

[1] A. Oliveira, “Configure a container to start automatically as a systemd service,”redhat.com,Feb. 21, 2023. [Online]. Available: https://www.redhat.com/sysadmin/container-systemd-persist-reboot [Accessed Feb. 27, 2023].

Posted on

I Survived Self-Hosting a Wiki With Podman!

Good Morning from my Robotics Lab! This is Shadow_8472, and today I am setting up not one, but two personal wikis on my home network. Let’s get started!

A Personal Wiki

Wikis are the reference material of choice for the casual researcher in this day and age. The content of the subject encyclopedia is turbocharged by the power of the hyperlink when compared to a volume/page reference that can take minutes to weeks to “load” depending on circumstantial accessibility. Community contributions allow for information to be updated in a timely manner, while built-in version control helps admins quickly repair sabotage.

This technology can easily be deployed to a closed-off environment for personal, group, or enterprise use. I know I could use one to organize my role play games, and my sister is after one to help with her writing. My goal for this month’s large project is to get both these wikis operational within our home network.

Wiki Planning

The first list of open source wiki software listed Wiki.js as supporting deployment on Docker. If my Rocky Linux 8 experience with ButtonMash has taught me anything, it’s that OCI containers are good for easy cleanup of botched installations, though challenges can arise when using Podman instead of Docker.

I spent a day studying Wiki.js off and on. My basic understanding is that you need three things for a wiki: the web server, the database, and the wiki software itself. I already understand a bit about the relationship between website and web server. Database vs. website is a similar relationship to website vs. browser. It is an independent process that serves data a website garnishes before presenting to a browser. It’s even possible for multiple websites to share a common database. While Wiki.js currently supports a few SQL (Structured Query Language) databases, PostgreSQL is the only database they will support in future versions.

Sparing a thought for my photo trunk project, I believe a wiki has potential for distribution once we learn more about how to use one. ButtonMash is configured to use the scanner though on its Debian install and not the Rocky 8 one, so I’ll need a different machine to host. GoldenOakLibry comes to mind as its primary function is to host and serve files.

My First Operable Prototype Wiki

GoldenOakLibry is a Network Attached Storage (NAS) by Synology running a custom version of Linux called Disk Station Manager. I found MediaWiki, the wiki software powering Wikipedia.org, in its package manager and chose the easy route. I did not know what I was doing for the most part – just that I was glad I re-enabled the old Vaultwarden container to store new passwords I made as I passed through Password Purgatory: database, wiki root, wiki admin (a user), database user – and then I had to make names for them all before I understood what each one did or how many more were needed.

There was a slight snag when the wiki wanted the database password and I wanted a new one, but someone blissfully using the same password for everything wouldn’t have noticed. A less tech savvy individual wouldn’t have thought to try looking for where to copy the wiki’s configuration file via command line. Once I figured that out though, I landed on a fresh wiki.

The snag that caught me was the mission-critical “What You See is What You Get” editor. Whenever I tried saving changes I made with it, it returned “[<RANDOM HEX NUMBER>] Caught exception of type Error.” A help topic on MediaWiki.org [1] reported fixing his wiki by installing a package called php7-zlib. This package is not in the Synology-approved repository, and I found no other package managers I’m familiar with when I connected over SSH. That’s… understandable, I suppose. The product is aimed at homes and small business too small for dedicated IT, after all.

An Alternative MediaWiki Host

A couple weeks ago, I had the misfortune of breaking one of the hooks securing the bezel around my laptop’s screen. Without it, I have to be extremely careful opening and closing the lid. I’m in the market for a new laptop, but in the meantime, the machine’s mind as it were is intact and I can’t use it for computing just anywhere anymore.

I learned a lot on my first successful prototype wiki. Database-website distinction and multi-site databases come to mind as relevant to my use-case. I’m imagining a system where I run each website in an OCI container with Podman on my laptop, then they go to a database on GoldenOakLibry for content.

…Podman isn’t in the Debian 10 repositories. There is a way to install it that involves a lot of hububaloo, but https://pkgs.org/ says it is on Debian 11, and I’ve had the computer upgrade bug as of late. My recent experience upgrading Mint primed me to locate a tutorial and upgrade to Debian 10. The process was the same (Timeshift, shift repositories, upgrade), just a bit less automated [2]. I took the opportunity to clean up after a failed project or two that involved repositories, but I think I ran into issues with Lutris’s repository GPG key (it updated later so I’m not sure). I’m leaving it for now.

The packages podman, cockpit, and cockpit-podman went on easily. Getting a static IP for the laptop was another story. Its official position within the house is under the TV, out of range of any free Ethernet cables we have laying about. After a few hours trying to understand how its Wi-Fi is even connected, I chose to move it next to ButtonMash and configure a static IP that way.

I started and enabled Cockpit with systemctl. It complained without a proper config file, but a browser on another computer made it to laptop’s Cockpit login screen. I told ButtonMash to link Cockpits, and it gave me a command I’ve been looking for for years.

ssh-keyscan -t ecdsa-sha2-nistp256 localhost | ssh-keygen -lf -

Admittedly, this only hints at a formula, but I saved it to a special directory on GoldenOakLibry anyway.

My Second Operable Prototype Wiki

With a more malleable host than GoldenOakLibry that wasn’t ButtonMash, I scrapped what I could of my first setup and started over. MediaWiki lists four packages as dependencies, and I removed three of them that related directly to serving web pages. MariaDB 10 stayed because I know for sure that it is compatible.

Unlike my experience with Rocky Linux 8, Podman on Debian 11 did not come with any unqualified registries configured, so I was getting fast searches with no results when pulling an image in Cockpit. I took a break for Sabbath, even though I felt I could keep the progress coming. When I got back, I about immediately found a tutorial that recommended a couple Red Hat container registries to add in addition to docker.io [3]. I spotted registry.centos.org in ButtonMash’s registries.conf; with the warnings in the file headers about who you trust, I removed it over the slim chance it gets compromised in the future. Worst case scenerio: I have to re-add it later.

Acquiring docker.io’s official image was easy next to telling MariaDB to let it in. I spent around seven hours inching through assorted tutorials tangential to setting up MediaWiki in a Podman container with [important keywords here:] remote access to a MariaDB database on a Synology device. It was slow, I could have written a post about just this paragraph, but I learned enough to understand the provided instructions (key tutorial: [4]). I braved Vim to write a needed config file and learned about MySQL database CLI client to make a pseudo-root account. And of course this was after locking things down to the static IP addresses I set up earlier.

Once MediaWiki was happy with its access to MariaDB, setup was similar to my first time, though I paid a little closer attention this time around and included all the editors, the mistake that send me on this side quest in the first place. The containerized setup will still come in handy, so it was not all for nothing. As a final, problematic sendoff, MediaWiki’s setup file, LocalSettings.php, remembers the port number it was installed to: future wiki installation attempts will happen in the containers they’re meant to run in, not some baseline I’ll be keeping around.

It was cause for celebration when I made the first edit and it stuck.

Project Notes

Given the right circumstances, I would have to say it’s possible for about anyone to bumble his or her way into a working self-hosted wiki on a Synology NAS, as I sort of did. Don’t get me wrong: even this is not an impatient beginner’s project! This week I learned that databases stand alongside websites, not inside them – a very important distinction for a sysadmin to know.

I’ve seen the Cockpit functionality to switch hosts since first installing Rocky 8 on ButtonMash. It was a pleasant surprise to find it worked over SSH and had a ready command for generating SSH host key fingerprints. DSM sadly does not have that functionality.

My opinion of Synology’s DSM began strong after a slow start, but it’s been fading. Stray one command outside their intended use case and it has DON’T TOUCH THAT! signs waiting everywhere. It’s still production grade, and that I can respect. I just won’t be asking for a similar system in the future.

The database password was extremely difficult to get right. No errors were ever thrown when entering 100+ character jibberish from Bitwarden, but 79 appears to be the maximum MySQL can swallow.

Takeaway

My progress this project does not represent a production-ready environment. I fully expect to have to tweak things before I have each wiki sequestered to its own user while still running happily. Website administration will be a whole other matter to conquer, but that is an exercise for another week.

Final Question

What kind of information might you organize with a wiki?

Works Cited

[1] Winel10, “Caught exception of type Error when saving changes in VisualEditor,”MediaWiki.org, Feb. 4, 2019 and June 8, 2022. [Online]. Available: https://www.mediawiki.org/wiki/Topic:Uuk96xjvh0ukaci2. [Accessed June 27, 2022].

[2] AM, “How to upgrade to Debian 11 from Debian 10,” AtechTown.com, 2022. [Online]. Available: https://www.atechtown.com/upgrade-debian-10-to-debian-11/. [Accessed June 27, 2022].

[3] J. Arthur, “How to Install Podman on Debian 11,” LinOxide.com, Sept. 20, 2021. [Online]. Available: https://linoxide.com/install-podman-on-debian/. [Accessed June 27, 2022].

[4] TechNotes “How to run Mariadb in Docker and Connect Remotely,”YouTube.com, Dec. 15, 2020. [Online]. Available: https://youtu.be/OabTOPOU2RU. [Accessed June 27, 2022].

Privacy Policy