Category: Networking

Good Morning from my Robotics Lab! This is Shadow_8472, and today, I’m assembling a NAS(Network Attached Storage) system. Let’s get started!

These things don’t come cheap. Between the case and four large hard disks, the whole system costs as much as a decent computer. It’s also worth noting that this is the first new system I’ve covered on this blog, aside from Raspberry Pis or similarly powered units.

In a way, it was actually a small blessing that this aspect of the project was delayed. In the time since I started researching NAS and now, Western Digital was found to be selling some of their archive quality drives as SMR (Shingled Magnetic Recording) instead of PMR (Perpendicular Magnetic Recording). This is bad because shingled drives are designed to overlap their magnetic tracks while writing so only the narrow read head can fit when the deed is done, and I intend to store more than just static data on here.

We bought four of the smallest non-shingled model. At seven Tb a pop, we should be able to dump all our existing data onto these things two or three times over should we see fit, but only after it’s set up, and that’s after reserving a full drive’s worth of data for parity.

Parity is a redundancy technique that provides some room for error. RAID 5 (Redundant Array of Independent Disks) does this by using a bit one one drive and adding all the bits in the same position on each of the other drives and ignoring all carries. It then distributes this parity data across all the drives, so the more drives in the array, the more efficient your data loss protection. If one drive is suddenly zapped out of [electronic] existence, you just need to replace it with another one of equal or greater storage capacity and the array can repair itself from there.

The NAS system we got was made by Synology –just to be clear, this is not a sponsorship, and I have never had a sponsorship– and I had some quality time with my father as we assembled it. Assembly wasn’t toolless, and the case was a little hard to get back on, but the photographic instructions were easy enough to understand, though they could have used to label their screws as being for either HDD or SSD.

Software wise… I’ll be spending another post setting things up, but I did take a self-guided tour of the place already. Synology has their own operating system called DMS (Disk Management System). I saw that and panicked. My shtick has been Linux, and at least trying to do it myself until stop learning and get frustrated. Long story short, I got bored and found myself reading the license agreement. I didn’t understand it all, but I found references to the GNU license in there.

According to one post on Redit I have since lost track of, people should be cautioned against toying around with different operating systems on this NAS solution because there’s a chance it could brick the system; I really don’t want that. Besides, it looks to me like DMS is built on at least the Linux kernel, and if that thread I found was to be trusted, it’s a stripped down version of Debian.

While writing this, I poked around in SSH and confirmed Linux, but determining parent distro is outside my abilities right now. It sure didn’t feel like a normal SSH experience. It dumped me straight into the root directory instead of a home folder –understandably without the disclaimer about free software and no warrenty– and when I got around to sudo whoami, it lectured me about the basics of superuser privileges, mirroring my earlier experiences in the graphical web interface.

From the moment I started installing the DMS operating system, I noticed how they somehow managed to design their user experience in such a way that anyone with basic computer literacy can use it, but they somehow managed to avoid insulting the intelligence of their power users. I didn’t need the utility they had to find the device’s local IP. I don’t need their online services to bypass port forwarding or their pictures and video galleries, but they’re there for people who want them.

My only substantial complaint so far is that the interface looks like it wants to be Windows, but clearly isn’t. The blue color scheme is there, but all the icons are off. I’m also constantly rummaging around through documentation for advanced features I don’t have a reason to know about yet, let alone disregard for the time being. I suppose I was a little miffed about GoldenOakLibrary being one character too long for the length cap at 15, but by removing a couple letters near the end, it’s still readable if you know what you’re looking at.

My big plans for this system at present are to make some sort of division between family photos, computer backups, and general storage. Once that’s done, we can finally start scanning, among other things.

Final Question: What systems have you used that don’t baby down their interfaces with scary warnings to the point of alienating power users who already knew what was going on?

Good Morning from my Robotics Lab! This is Shadow_8472, and today, I am getting a head start on this post, so I don’t yet know where the end point for this segments. Let’s get started!

I hate networking. This router project is really taking a lot to learn. I’m losing track in my memory about what point I was on when the official break in weeks happened. Right now, I’m just looking forward to the time when I can say I have a custom router, and setting up the “Beowulf Cluster” I originally set out to build –not that I can really call it that anymore– should be one more week™ after that.

Once I had the network interfaces set up correctly, DHCP wasn’t far behind. I had a few semicolons missing after some lines, but that was about it. I plugged my laptop into the my little subnet and the Pi assigned it an IP in the range I had told it to. It just wouldn’t let me at the Internet at large, even though I haven’t changed iptables off its default of accepting everything yet. Of note: I was unable to use the Internet even while still connected wirelessly. Two weeks ago (Was it only that short ago?) I would have just poked around with “black box reasoning” before concluding that Ethernet superseded Wi-Fi because it is faster, all other factors being equal.

Further investigation led to believe I should look into NAT (Network Address Translation). It’s disabled by default, likely because if you need it, you’ll soon learn about it like I’m doing now.

A couple guides later, and I had uncommented a line in a config file to allow IP forwarding, but when I used modprobe ip_forward to add NAT abilities to the kernel, listing out my iptables gave a warning about legacy software.

***

Well, this write up is coming due, so here’s what I’ve got.

I worked with a couple friends on my private Minecraft server –yes the one I’ve covered here before– by the names of doflagie and TheRSC. We didn’t go messing around with NAT tables, but we did inspect the DHCP configuration files some more.

At least, that was the plan at first. I accidentally broke an interface by editing the file there. Rebooting cut off my connection. I SSH’ed in over the other interface and promptly broke that one too. I had to swap around cables to fix those config files, so it was all good.

With my little review from last week out of the way, I went to weed through the actual DHCP config file. I wasn’t sure where it was, but before I resorted to finding it again online, I noticed one terminal on my workstation was already in a local DHCP directory when it wasn’t SSH’ed into somewhere else.

As part of my workflow, I hooked BlinkyPie up to the Ethernet subnet. Thanks to the default Raspian settings, eth0 overrode wlan0 as the default network interface, so I didn’t have to disable or disconnect Wi-Fi. Otherwise, Blinky is standing in for a generic microcomputer on this micronetwork.

It took several hours, but when I rearranged the DHCP config on the Pi4 to reflect the subnet and not the wider home network, I was able to reach out from Blinky through the Pi4. However, I was only allowed to the gateway, and no farther.

These results are underwhelming. Each last problem feels like it is comprised of a chain of smaller problems, leading to another such chain. I’m making progress, but if feels like the goalpost is moving on me.

Final question: How hard should this really be? Sure, there are higher lever programs, but how long should a low-level approach like what I’m doing take to figure out?

Good Morning from my Robotics Lab! This is Shadow_8472 and today, I am working on that Pi4 network piece again. Let’s get started!

I am having the hardest time with networking. What could have been a confusing, possibly straightforward black box operation turned into a subject of study where I’m likely going to be able to do my own thing later on. With that better foundation, I just might add DHCP back in into my general plan, as such a system would be a much more useful tool I could use in an upcoming project I’ve been looking forward to.

As stated in an earlier post, I’ve been cobbling my way together using a series of incomplete tutorials and seeking help understanding the gaps. Many video tutorials for networking are either old, made with bad audio quality, have a presenter with a thick accent I can’t make out, or a combination of the three. Text is unfortunately the way to go.

My functional goal for this week was to get the Pi 4 functioning normally online with a static IP. Through whatever mishmash of tutorials, the following is what I’ve cobbled together as I understand it.

The Pi 4 has two network interfaces, one called eth0 and another called wlan0. These names represent an old standard on the way out, but for this week at least, I’m sticking with them. I started by going to /etc/network/interfaces and set it up so each interface has its own configuration file in the directory /etc/network/interfaces.d/. Eth0 was straightforward to set to static, but wlan0 had another pointer elsewhere for the WiFi name and passkey.

My first big problem was in how I had to copy configuration file examples by hand. At my worst point, I had some equal signs where spaces belonged and spaces where underscores belonged. The former was a mistake I made when looking at two related, but dissimilar files, and I diagnosed it by paying attention to the error messages when I tried sending sudo ifup wlan0.

The later was a mistake I made because I was copying things by hand without SSH and a dark mode plugin I use actually cut off the bottom row of each line of test in a code box, including the entirety of each underscore. This one only told me the interface failed to start. Someone going by Arch on the EngineerMan Discord pointed me to journalctl, some sort of log viewing utility. I piped the wall of text through grep to pick up wpa_supplicant, a utility that kept complaining. Journalctl also pointed out locations of errors with quotes around the WiFi SSID and psk fields where I needed strings.

Assorted errors along the way included a mistaken time zone that was easily taken care of. I also had a surprise when the contents of /etc/wpa_supplicant/wpa_supplicant.conf vanished on a reboot: turns out I had a blank copy in /boot and as Raspbian boots, it will move certain configuration files in that folder around to overwrite their permanent counterparts, leaving me guessing as to what’s already been done.

With stable network interfaces, I started testing them, but I wasn’t able to ping anywhere successfully. I noticed that it was using its IP from the eth0 socket, and not wlan0 for some reason. Eventually, I either found or was directed to route or route -n: basically the table where Linux decides what door to send packets out of as they leave. Eth0 was set up as default if no other matches on the table were found; all my ping packets were being sent inward and getting lost instead of getting sent out, into the wild.

I spent time studying this new table and found a clear, black box style tutorial I was able to reverse engineer LINK. Manipulating the table directly was straightforward enough, but what I failed to realize was that parts of the table are generated fresh each time a network interface goes down or comes up. My work totally collapsed when I rebooted, reverting the default interface back from wlan0 to eth0.

The fix (shown as “Part 2” in the black box tutorial linked above) led me back around to the beginning. Part of the configuration of different interfaces includes a setting called gateway. I actually had to move it from eth0’s config file over to wlan0’s config file.

Admittedly, I am a bit disappointed in this week’s results, but I learned a lot more than a total black box experience would have afforded me.

Final Question: Have you ever poked at a “black box” and learned how it worked?

Good Morning from my Robotics Lab! This is Shadow_8472, and today, I actually spent the better part of the week on this segment and still didn’t get as far as I would have liked. Let’s get started!

There is a reason networking scared me off when I first looked into it back when I was in university. Every computer needs a way to introduce itself to not only its communication counterpart, but all the message carriers along the way. It’s almost unbelievable how easily it all can be smoothed over so even a toddler can get him or herself into trouble without proper supervision. But all the inner workings are still there, churning away, ready for someone to go monkeying around until something breaks.

Last week, I talked about the hardware side of things. I have all the equipment I need ready to use, but it’s waiting on a digital support structure to hold the worker nodes together. My goal for this week was to build that support structure. I would have declared success when I could connect my laptop to an Ethernet network served by the new Pi 4 and access a web page.

Originally, I was aiming to set the router up as a DHCP server so it could hand out IP addresses on the fly. Along the way, I learned how to set up a static IP. Did I mention how ugly raw networking configuration looks to complete beginners?

I honestly don’t know half the stuff I did as I was trying to follow multiple tutorials at once, hoping one would fill in for another when it didn’t apply to my present situation. One blank on a text-based tutorial was elaborated on by a quick comment by a YouTube video, and I ended up stumbling in a third direction in the hopes I was doing the right thing for my project.

And that’s when I can even follow the tutorials! Half the time I landed on a video where some would-be instructor may know what he’s talking about, but his audio quality says he’s working with what he’s got, and that can be a bit sharp on the ears. Another good chunk of the time, it’s someone with a thick accent I wouldn’t be able to understand unless I could ask him to repeat multiple words. Unfortunately, text is going to be the way to go for tutorials at my level of specialization.

I wasn’t able to get the DHCP server sorted out. Something about there can only be one per network, and I have no idea how to confine it to confine it to one network interface — that’s another thing I learned about.

Network interfaces: the whole reason this project is a thing is because the Pi 4 has two separate network interfaces, or ways to access a network, such that it can participate in two distinct networks at once: one for the Ethernet, called Eth0, and one for the WiFi, called wlan0. If that weren’t confusing enough, there’s a new standard that names network interfaces off the hardware they’re connected to. For my intents and purposes, where I only have a single interface per kind of connection, it only serves to confuse when it shows up elsewhere — namely my laptop — but I imagine it would make handling a 50 port switch a little more bearable.

At some point, I gave myself a reality check. I only had five computers to contend with. DHCP is a little bit much for what I’m looking to do. Extra overhead: needs to be axed. I’ll just use static IP’s all around. I went ahead and set my laptop’s Ethernet network interface up with one and rebooted, since I couldn’t seem to restart just the network settings. I had to fix a couple settings, but it was pretty easy to get it to ping –and even ssh into– the Pi 4 over the Ethernet network I had going. The trouble was that I couldn’t access a web page like I wanted.

I messed around a bit and when I was done, I unplugged and connected to my home network, only to find it was being difficult. The best I can figure is that by editing one of the configuration files to give me a static IP, I was accidentally voiding some default for WiFi settings as well. I made an attempt to copy what I had going on from a tutorial on the Pi 4, but due to a series of disconnection issues, I ended up restarting the Pi and losing it on the network.

I was sure to make notes on how to reverse the network damage to my laptop as I made it. I commented out the altered lines, and rebooted again. Still no Pi 4 on the network. I looked on the miniature network again, and eventually resolved to swapping over the HDMI cable to the back of the monitor, only to find a line in ITS network configuration files from when I was experimenting with iptables to solve the present issue.

I’m laying this issue to rest where it is for now. I had some measure of tangible success, but the majority of my efforts were sunk into learning new skills. My laptop is in a usable state, at least.

Final Question: What have you cut out to make things easier on yourself?

Good Morning from my Robotics Lab! This is Shadow_8472 and today, I learned a lot toward the next step in my project and backed out at the last minute. Let’s get started!

You know the thing about clusters: it takes some overhead to run them? The computers I’m working with in my Third Workshop care package don ‘t have much CPU to begin with. Special thanks to kevrocks67 on the EngineerMan Discord server for getting that point through my head and pointing me in the right direction: subnetting.

My understanding of subnetting is lacking to say the least. Perhaps if I back up and just tell the story.

This week saw most of the parts I need for my cluster come in. Shopping for the Pi 4, I compared different sellers and chose one to go along with a separate heat sink case I picked out from a different distributor. I managed to save a little money by assembling my own kit instead of getting one that included some small, aluminum heat sinks.

When the case arrived, it came with some thick thermal pads. Yeah, those won’t be separating once the case is fully installed and operational temperatures melt the pads into place or they are just sticky. I did a dry fit and it doesn’t look like the case will operate correctly without them. In the meantime, I noticed the camera cable slot and the similar-looking display slot will likely be inaccessible once I assemble the case, though the GPIO pins will still be open.

My father picked up a couple outdoor grade 3-way power spliters for power management. The minicomputers I’m working with have bulky transformers right on the plug, and they are such a shape as to make it difficult to creatively arrange more than three on a seven-plug power strip.

The Ethernet cable we picked up was paid for with a little effort. One of my church members was wiring up the church a while back for Cat6, and he has some left on the spool. I stayed topside while my father donned a jumpsuit and finished running the cable to just below the office, where he cut the cable. There was enough to reach the office, but that was it: maybe about 30 feet max: more than double my very rough estimation of what I’d need. A package of Cat6 ends and a crimper will see me learning yet another new skill in the near future.

About the only piece I don’t have picked out yet is a board to mount things on. It doesn’t look like the heat sink case has any good places to mount, though the switch and minicomputers do.

The next logical thing to do is assemble an early prototype! One control node and one worker node should be a good start… if only I could get the pi to boot from an SD.

Lacking the desire to format any drives where a valued drive can possibly be hit in the crossfire, paired with the inability to read SD cards on anything but my laptop, I tried using an app on my tablet to flash a card. For one reason or another, I wasn’t able to boot. I suspect I need root privileges to properly format anything with Android, unless I want to format as internal or removable storage. Lacking access to an old phone nobody has no qualms with me “voiding the warranty,” a multi-SD reader is on order.

Speaking of SD readers, turns out each of the five little helpers has one built in. I’ll be using one of those along with a live USB for formatting fun in the near future with no chance of fireworks at the end, just as soon as I figure out why they look exactly the same with lsblk -l. I suspect it’s got its own SD card like storage device in there. Ironically enough, I suppose the difference should make a slip up of the same calibur nearly impossible this go around. Nevertheless, I still want to practice isolation of formatting whenever possible. In either case, what I am trying to think of as equivalent to a hard drive is not labeled sdx anymore.

Somewhere, I made the switch in short-term project aim from cluster to subnet to reclaim some of that overhead. I still want to end up with a cluster for the experience, but for now my token effort among token efforts will be more efficient with less overhead.

To help with overhead costs, I found a site that compares lightweight distros. Thishosting.rocks/best-lightweight-linux-distros/ They have a sortable chart of about 30 distros and their rough minimum system requirements, including RAM, CPU, and space on disk. Folding@Home is likely to use CPU above anything else, I suppose, and RAM can’t be easily expanded, so my dump stat is disk space.

I optimized for CPU, and found Debian was a near-perfect choice. Most, if not all, my Linux experience is in Debian or a decedent of Debian. The required RAM is one of the lower rankings while still needing an almost non-existent CPU. I even already have the install media plugged into the thing and booted! Though I may want to look into a server installation option.

In terms of mentally tangible progress toward a prototype, I set the Pi 4 aside for the moment and brought in BlinkyPie, my Pi 3B+. My goal: turn a Raspberry Pi into a router. I’ll know everything works when I have Internet access on the minicomputer that is otherwise not connected through any other means.

I viewed a few networking videos that were either dated, covering basic material I already knew, had bad audio quality, or showed what to do without giving any theory behind their actions. I eventually settled on a clear, step by step tutorial on at Linuxhint.com/raspberry_pi_wired_router that does not explain its actions, but the steps are short enough that I can seek out my own explanations as I go.

The tutorial starts with installing Raspian and getting SSH going. As anyone who’s been following my blog long term knows, SSH was a real pain for me to figure out, and if I didn’t know what I was doing, it might as well be a small element of a magic ritual to start over if things fail. Since I intended to work with a Pi that was already established, I glazed over those steps and started in on configuring the network.

The whole reason this project of a subnet is possible with the hardware I have is because the Pi can participate in two networks at once: one via Ethernet, and one via WiFi. To make a long story short: this procedure requires the Pi establish itself in a special IP address for the router, implying a static IP. I sometimes take this Pi elsewhere (or at least its chip), and I don’t want to forget and accidentally mess up someone’s network, so I backed out in favor of waiting on a properly running Pi 4.

As always, this project is evolving as I work on it. The Gigabit Ethernet speeds I thought I would use for a cluster won’t be necessary for the time being, as there’s a slightly lower bottleneck at the Pi’s WiFi network adapter. Next week should be interesting.

Final Question: When was your last project where you lowered the specs and found yourself overbuilt?