Good Morning from my Robotics Lab! This is Shadow_8472, and my main tower broke last night, so I’m getting a head start on this week’s post. So today, I’m teaching myself how to fix it. Let’s get started!
The Error
I wish I was writing this week about my experience getting Space Engineers to run in WINE. I had spent my Monday in another game only for it to glitch and lose a bunch of stuff. I needed a break, but not like this:
Warning: /lib/modules/5.10.2-2-MANJARO/modules.devname not found – ignoring
mount: /new_root: unknown filesystem type ‘ext4’.
You are now being dropped into an emergency shell.
sh: can’t access tty; job control turned off
[rootfs ]# _
A lack of keyboard response wasn’t encouraging. Neither was a lack of a response from my usual Linux help places. Still, it’s polite to at least drop the error off for an Internet search, but that is a lot of information to throw around. Any relevant results I got were obscured by my inability to understand them.
As if I weren’t having a bad enough day, I went to try booting to an install disk, thinking it to be for Manjaro. It was for Ubuntu — and it was corrupt. All I saw was what looked like a spaghettified terminal in the pattern glitched spots on the screen were appearing.
Understanding the Error
I tried finding my GRUB disk, but it’s lost. After eliminating bad USB ports as the cause of the corruption, I burned a new Manjaro drive over Ubuntu using my Manjaro ARM card in the Pi 400.
Line-by-line Internet searching for the error didn’t get me much farther than before, but breaking the first line apart was the key because it contains a file path. I booted to my newly reeducated drive and mounted my file system to look around.
/lib/modules/5.10.2-2-MANJARO/modules.devname sounds like a file path, so that was a good place to start. Sure enough, no such file exists, but there is a modules.devname.old file. Looking one directory up, there are a few other similarly named directories in a similar format — the next most recent being 5.9.16-1-MANJARO. It did contain a modules.devname. I considered renaming the old version or copying in the previous one, but I’d have had no idea what I was doing, and there were some slight differences I later found by overlapping transparent terminals — not just side-by-side comparison.
I already knew Manjaro has the ability to install and switch between multiple kernels, but I had no idea where they’re kept or how they work. Using this new information, I looked up the names, and I’m reasonably certain I ended up with a bad kernel on my system.
Fixing the Problem
Now that I had a lead on what to pay attention to in search results, I learned that people are complaining a lot about this and similar kernels released around New Year’s. I would like to try reverting a single kernel and seeing how that works.
One problem: every guide I’m finding appears to assume I’ve already booted to Manjaro and gives either a GUI or command line options for reverting.
In my experience to date, I would have expected I could just go in, flip some strings in config files around and call it a day. It appears I have no such luxury though. That or the kernel is so central to the system that “luxury” is not having to flip 45 such config files, not all of which are meant to be human readable (I have no idea here). Nevertheless, the tool of choice appears to be a command called chroot.
Chroot –as I understand it– is essentially used to CHange ROOT (the directory, not the superuser). It opens a new shell with a root directory of your choosing within reason. It can be used to repair broken operating systems using an external kernel, but there’s a similar concept I am not covering today called chroot jail where users accessing a system from the outside are only shown a subset of the system’s actual root directory.
For this post, I will be using the guide “How to: Chroot into a broken system via live CD/ISO or alternate Linux system” on the turnkeylinux.org to set up a chroot and the command line instructions from “How to Switch Kernels in Manjaro Linux” to revert my kernel.
Side note: a different guide than linked brought up a program called os-prober. I found it in both Manjaro and Debian. When I ran it on my tower, it quickly found my main Manjaro install, but didn’t find my Windows drive I believe to be connected but not mounted. Other methods showed me evidence that it’s still there.
Attempt 1
I didn’t completely follow the set of instructions on how to set up the chroot. I performed them from my root directory ‘/‘. Things started going wrong when my second chosen set of instructions weren’t clear at all about how to manage multiple kernels from the command line.
I started improvising. I tried to launch the GUI version of systemsettings5. Going through the menu pointed straight to the live USB’s kernel, and all its settings no doubt. Going through the command line was fruitless until I had exited out of the chroot and umounted all extra directories, even when I used su shadow8472 to switch to my normal account, a procedure which root could not reproduce after the chroot ended. In short: I did something right, but I’m not done yet.
Attempt 2
This time, my plans required access to the terminal. My plan was to proceed with the same chroot setup, then work my way into a graphical interface hosted with enough elements from my actual tower to work. Any attempts to use a shortcut using Ctrl+Alt+<function keys> only switch to close representations of the true console, but in reality are running on top of that x server I needed to free up. I followed another set of instructions found here, set up the chroot, and it failed.
Third Try
Left with no other options I begin to understand, I moved on with the command line method of actually uninstalling a kernel. I received a stiff warning about how removing the suspect kernel, linux510, breaks a dependency required by linux-latest.
This attempt was interrupted by someone who sounded like he knew what he was talking about suggesting a few things.
The Actual Solution
Changing direction again, I found this Reddit thread that showed me how to expose a GRUB menu for which kernel to boot. From there, I was able to boot a working kernel and get in.
Out of curiosity, I did try to remove the bad kernel from the GUI tool, and it appeared to work until it was still there after a successful report. I was able to bring up the same failure message from the command line in a dialogue box.
Minor Developments
I’ve had tons of issues with my Wi-Fi card, so I turned it off and this week while focused on it, I’ve opened the case, cleaned it, and pulled the card. Maybe it’s software, maybe it’s hardware, but as of this week, I’m using my Pi 400 as a Wi-Fi to Ethernet receiver full time until a suitable replacement can be arranged.
Takeaway
I never realized how far I could get myself sans help on a problem I’ve never seen before. Perhaps I’ve reached a point where my skills are not readily surpassed by those around me. The one recommendation from a person who sounded like he knew what he was talking about turned out to be a dead end (removing the .old extensions from all the kernel modules).
Computer problems can happen to anyone. They’re just easier to learn how to fix without special training on Linux.
Final Question
When was the first time you solved a complex problem you had never encountered before without help?