Home lab. How to build your dream home infrastructure
Take a ready-made boxed opensource solution (examples TrueNas, ESXi Free/ProxMox, OpenMediaVault, etc.) and build your project on it
Or take a bare OS (from FreeBSD to Alma/Rocky in our time), and assemble something of your own from packages and opensource applications
The second point is taken from a quote from a colleague who, in general, said so. At that moment, my knowledge and experience were not enough, so I understood his words on how to take a conditional python and write your own gui. At that moment I thought that only gods could program.
As a rule, the second way will not have a ready-made and beautiful “GUY” in order to solve all problems and settings using buttons, and, again, as a rule:
The first time the solution will be through Google + terminal commands
When repeating the steps from point one for the second time, you will want to turn them into a script so as not to repeat commands (or remember)
The third time, taking into account that there may be different parameters and variables, the scripts will become interactive, consider the terminal interface of your self-assembly
And fourth, if you’re lucky, over time a ready-made container or package will appear or be found that will contain everything you need to manage your self-collection
Well, don’t forget that all open source “boxes” or even commercial ones are essentially assembled Linux, with some set of open packages, where the only proprietary package is a written GUI for the solution. Well, okay, a set of settings and optimizations so that the “box” works better than Linux with the same packages.
Next, a little basic theory and history to understand what the final solution from the equipment or hardware side would look like if we were a YouTube blogger or an IT enthusiast with a lot of “useless money”.
Let's look at the diagram from above. It was written entirely by me based on what I observed in practice. It is important to note that it is superficial, that is, without details and nuances. And, of course, if we are building a full-fledged cloud, then the schemes will be clearly different. Here we have collected almost all levels for our data center:
network equipment for connecting everything to each other in a fault-tolerant format
two active computing servers (VMs) with a third, which will be a backup in case one of the two active ones fails
two storage servers, in an active-passive format, where one collects data on itself and then replicates it to the second for safety
Why is that?
Well, with network equipment it’s clear that if one breaks, everything continues to work because there is a second one. And the broken one is replaced, under warranty or not.
The computing layer, or Compute Server, will always need to be expanded as needs/business grow, so it’s worth following the 2+1 scheme: two active ones for one backup, so the computing power is spread between two servers for performance and there is room to move , if the RAM or another component of one server has become bad (not completely crashed, but partially unavailable).
We will also place the storage equipment in a separate layer, so as not to burden the computer part with the storage load, and “not to keep all your eggs in one big basket.” We have a 1 to 1 ratio so that we have a full-fledged replica of our data, because if, conditionally, fewer clients visit us while one of the host computers is broken, we will earn less, but the business will live. If we lose all the data, then we can assume that there is no business (here we will include the situation of data recollection when one of the disks falls out, the array will work obviously slower, which will affect the business one way or another).
Why do we need such a “stuffy” theory? The experience of trying to build my own self-sufficient infrastructure at home led me from a home NAS, not a Celeron, to essentially the same scheme on “minimum wages”.
She looks like this:
Let's talk a little about her, and why this happened:
Previously, in place of the router I had Mikrotik hap_ac2. As my network at home grew, this simple version, designed to simply route and distribute WiFi, was no longer enough.
Also, Mikrotik played the role of CapsMan, which manages two access points (APs), which created its own additional. load
Well, Mikrot was also a kind of switch at one time, which also did not allow him to live in peace
In this scheme, the role of microt will be played by a Mini PC from Alishka with 4 network interfaces. We will deploy a software firewall/router on it. Nearby on the VM there is also Mikrotik CHR for CapsMan. Upstream cable to the operator, and another cable to the POE Switch.
Our POE Switch will power two access points at once (previously this was done through microt adapters, and it actually looks very poor, even though it’s hidden in the closet).
Through beautifully laid cables in the apartment, power will go to the access points and from one cable to the beam where the main “machines” are located.
Gigabit switch for transmitting “Internet” via cable to all devices on the balcony: Thunderbolt dock, gaming PC, mini-PC computer, storage server
A 10-Gigabit switch will live on a separate network (VLAN) only for storage for: MiniPC, gaming computer and the storage server itself.
What we will come to in the last chapter of this story will look something like this in the second diagram.
Looking at it and removing the local jokes there, you can see that the diagram of two switches with separated loads (Internet traffic and storage), with a separate machine for each level, brings us closer to the ideal scheme of a near-building local data center.
Part one. From NAS to first home server
I learned about NAS at my first job. There, the business was built around backup solutions for both corporate infrastructure (bases, VMs, hyper settings) and PC backup solutions (essentially individuals, or later on consumers).
Well, it’s obvious that the second category of users used NAS from different manufacturers as storage for their data, such as Synology, D-Link (if I’m not mistaken here), the conventional Apple AirCapsule and others.
Then a man came to our work who was clearly infected with Hyper-V fever from Microsoft, and he told how, on old hardware (from one of the old PCs), he was able to assemble a Hyper-V server with a bunch of disks, which he gave to the Xpenology VM (hacked Synology) and how cool this whole thing is.
If we leave the addiction with Hyper-V, then the thought of our own home storage seemed quite tempting:
at that time I only had my own Mac laptop and gaming PC
the archive of photos, videos and other important content such as documents was inside this PC on a separate fat disk, and if something happened to the OS of this PC (viruses, encryptors that were popular at that time), then essentially goodbye to the past.
a gaming computer as the only equipment with open access for me, there were also all sorts of Playground for the first acquaintance with Linux, attempts to install MacOS on a regular PC, etc.
Well, obviously examples of problems, this is what I experienced when, by the sweat of my brow, I tried to recover a 4 TB disk with a bunch of photos of my little brother and sister using all sorts of Recovery tools after a bad experience with Linux
The idea began to languish in me and took off after I started a new job (where I had a salary x2 from the previous one), which gave me the opportunity to spend some money on a new gaming computer (only with an SSD) and separately on a Mini PC for storage .
Why a MiniPC and not a ready-made solution from Synology? Well, first of all, there was information that Synology can be run on anything, and, accordingly, the machine can be prepared better than the top-end Synology was sold in the store.
Then, of course, the price: the same percentage, but with x2 RAM, 1 SSD for the OS and a new 4 TB disk (for an existing mirror) it turned out cheaper than the average ready-made device.
In fact, the sum of the parts came out to 22,300, while the Synology analogue (without disk and frame) was approximately 18-20 thousand.
We've decided on the hardware, but we probably need to mark something:
everything is clear in the case, I bought the cheapest one + with a power supply and the main thing is that it can accommodate 1 disk for 2 with a floor, and 2 disks for 3 with a floor
RAM too, from a good manufacturer but the cheapest. In terms of speed and size, the maximum that the processor can handle.
It’s clear with the disk, 4TB for the mirror I already had 4TB.
250 SSD for the OS, for a reason. The logic is that for an OS like Linux, 64 is enough, 128 is enough with a margin. But 250 was chosen for a reason, and then it will become clear why.
Well, a motherboard with an integrated processor. Firstly, all sorts of Celerons for assemblies are supplied only this way, namely this mother and father were chosen: The processor at that time is the top-end Celeron on the market with support for 4k through an integrated video card. It had 4-6 Sata slots, which was ok for expansions. And a small PCI connector for a second network card up to 2.5 gig (but then it was just space)
Why did I so easily agree to such a scale of disaster?
Even despite the presence of Netflix and Amazon, there was still a problem with content in the Russian Federation. If we could watch American cable TV series on Netflix, it happened six months a year after the start of the new season (after its completion). Also, we never had normal access to Disney+, and good content was just starting to come out there.
I have some kind of automatic dislike for pirated online cinemas, and I especially get burned by unexpected advertising in the middle of an episode, after which I definitely need to go change my panties, or “sports betting” plays in my head for a week. To be honest, the quality and stability of the video players in these cinemas, and the storage back that they have, are not taken out to receive high-quality content on their 4k TV or iPhone.
Then I learned about a “sign from above” called Plex (Plex Media Server), which allowed you to make your own, conditional, Netflix for yourself from all the content that you have on disk (downloaded or purchased). But such a solution to host on your own computer would be redundant, and all NASs from Synology and others even had an official server application.
And of course, since my gaming computer was entirely on an SSD, downloading torrents onto it was not an option, so a separate mini PC with an application for downloading torrents and regular disks would be very cool.
Total:
downloading content and torrents
broadcasting content from your equipment to your own device (at home and outside)
storage of archives and important data
and just space to save PC backups before different tests
These 4 reasons made me order the previously listed equipment from an online store. Next, the task was to figure out what to do with the Software part of the new solution.
Since my first work was heavily related to VMware ESXi client, I did not see a better solution than deploying it on my computer. Why hypervisor? XPenology had some problems with the equipment I chose, so the problem could only be solved through hyper. Why XPenology? Stories from a colleague at work, and experiences from what I heard from clients.
although already at that moment there was a better solution, which we will come to in the end
Well, a fairly rich library of applications for iOS: chat, media, photos, etc., etc.
So the software plan for the new piece of hardware:
We deploy ESXi to a 250 GB disk and create a datastore for our only VM there
Deploying a VM from the XPenology image for VMware
Using PhotonOS scripts, we set up a direct link from Var’s “virtual disk” to a physical disk
Setting up DSM, enabling torrents, plexes, smb and downloading data
Which, in general, is what he did.
What happened next?
Operation of the solution was quite boring at the beginning:
Activating branded applications with an unlicensed server required effort, but there were instructions
Launching and installing left-field applications (for example, VMware Tools for DSM) also required dancing, but everything was solvable
I tried Plex for the first time there, but there was a feeling that something was wrong
And then something terrible happens: just like that, while copying a file, the Synologe mirror falls apart. I, as an inexperienced engineer, immediately started looking in my wallet and on sites where I could buy a new drive at a profit, but still, having tried to “reassemble” the broken array, the operation went well and showed that everything was ok.
I didn’t find a proper answer to the question “why”, so I began to come up with holes or nuances of my solution myself in order to find an “improvement” for them. And then by chance I remember that when I was playing around with Linux and a VM with Windows on it, there was such a topic as Passthrough, when you don’t map one device to another, but rather forward an entire PCI device inside the OS VM.
After googling, I found out that I have two Sata controllers for 4 Sata ports, and one can safely be used to “hardwire” the Xpenology VM disks. Only after such a feint with the ears did the performance become even worse.
Having returned everything back, I decided not to overload my machine for now and moved on with my life.
I was lying when I said that only everyday reasons became the beginning of my journey to the home lab. The main reason, as it now seems, was the desire to be like my idol at that time, Linus from LinusTechTips, whose videos at that moment were about how to assemble your own NAS from improvised means.
After watching another portion of Linux, I learned about the existence of ZFS, OpenZFS in variations for all Linux platforms. And taking into account that this was a ready-made package and solution for managing disks and files within the OS (namely, the lack of understanding of mdadm at that moment did not allow me to immediately switch to Linux and build storage with my own hands), I decided to “migrate” from Xpenology/ Synology on ZFS.
At that moment, of the packaged solutions, there was only FreeNAS, which was not very clear to me at that moment, and also on FreeBSD, about which I only knew that I needed to “patch KDE for FreeBSD” (and generally not understanding that every word means). By that time, I already had good practice with CentOS, and I was familiar with Ubuntu from the time of tests on my computer.
It was decided to create a second VM on my Ubuntu-based mini PC. Transfer the data to temporary storage, rebuild the ZFS inside Ubuntu and upload the data there. I made my first attempt by forwarding the device's PSI.
I wanted to leave ESXi so that there would be space to create some kind of temporary VM for a crappy test, and not kill my “gaming PC”
But just like with XPenology, there were problems with forwarding performance, and the GPU was not accessible to the VM.
The beauty was that with such forwarding and without mapping virtual. disk to hardware, I was able to simply install Ubuntu instead of ESXi and import my storage.
Which is what I did.
Plex installed from the developer's turnip, found a cockpit, some GUI for managing Linux. Next, I installed a set of plugins for managing ZFS and simple disks (such as SMART) to the cockpit, thanks to 45Drives. Transmission straight from Canonical Turnip.
Then a problem arose that I had to subscribe to TG groups of online cinemas, and use notifications to track the release of new episodes of TV series, and manually add a download tracker to the torrent through the browser. And therefore, clean everything by hand. Horror.
A little Google introduced me to the solutions of the Servarr team: Lidarr, Sonarr, Radarr – each of the containers is responsible for its own type of content (music, TV series and movies). You can add works or groups that interest you, and the application itself will then parse torrent trackers, find the required download (you can specify maximum quality, voice acting, etc.) and add it to the torrent client. And then, when you want to get rid of what you watched, it will be deleted from the hard drive.
Installing the solutions looked tough, but the alternative installation via a Docker container seemed quite simple and convenient. A Bash script with a series of docker runs and mounts built an entire solution for automating content retrieval for me in a couple of minutes. Then, by chance, I found out that both Plex and Transmission can also be run in Docker, and thus described the ideal software to launch when moving. I moved them too.
For Servarrs to work, a couple more containers were needed (the torrent tracker parser itself, which transferred data, solutions to all sorts of challenges, and other service containers).
At this point, I didn't know that Plex was using the integrated GPU to play 4K content. And when it was a package in Linux it had direct access, but for a container through Docker, it was necessary to mount the device and other things. I became acquainted with this part very late. I emphasize this because this will be one of the reasons for the upgrade.
And in fact, life began when storage worked completely on its own in the context of uploading content to Plex. The SMB share, accessible from the local network, was used to upload archives and backups. All sorts of new jokes and wishes were looked for on the Docker Hub and added to the monstrous bash script at that time.
Yes, yes, I’ll find out about docker-compose myself later
But the transition of almost all content producers to 4K began to not allow me to watch everything without “downloads”, and over time the free space became noticeably less and less (and we remember that the case could accommodate UP TO two 3.5 disks).
