/cloud-machine

My Personal Cloud.

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

cloud-machine

cloud-machine is a set of scripts that uses beanstalkd and vm-bhyve to create virtual machines with FreeBSD as guest OS (with the latest patches installed) in a distributed fashion. Some customizable scripts also install Tailscale, AppJail, DNSMasq, Director, WireGuard and more, all ready to use.

Architecture

The manager (user who administers the cluster) on one of the cluster hosts (in this case srv001 and srv002) sends, using beanstalkd, a job to create a new VM with all the parameters specified by the manager. The manager can be on the host to which the job is sent, so it can be source and destination at the same time. Although the manager can submit a job to another host in the cluster, if it continues to submit jobs to the same host where it is located and reaches the defined limits, the job is redirected to the next host and the same decisions are made if the limits are reached. The final destination of these "hops" is when the "next host" is null, so there is no possibility of creating a new VM. This scales very well without adding more layers of complexity.

After a host picks up a job, it will create the VM. The instructions for creating a VM are in the create.sh script, which is the same for each new VM to be created, but what may be different (and what gives you the power) are pre.sh, local.sh and post.sh, the customizable scripts that run only if they exist.

If you want to learn more about the responsibility of these scripts, read the Files section. But I will explain at least some relevant responsibilities that create the architecture you see above.

The customizable scripts configure more things than create.sh as I said, but the important thing is that they configure Tailscale and WireGuard. Tailscale is used to communicate with the user globally in a secure way. Tailscale allows you to use your exposed services from your VM to any device on your Tailnet. WireGuard is used as a way to communicate between the same VMs on the same VPN. The performance with WireGuard is of course better, but more importantly it can allow two or more VMs to communicate with each other even if they are not owned by the same user.

Once the virtual machine is up and running, it has AppJail installed (and configured), Director, DNSMasq installed (and configured), WireGuard installed (and configured), Tailscale installed (and configured) and a few more tools, such as cpdup, which is used by LittleJet.

Goals

This is an internal project to deploy virtual machines using vm-bhyve around all my machines even if they are not physically in the same location. I have found that I can designate certain responsibilities or tasks to specific virtual machines to get a better organization of all my systems. This is of course motivated to use my other project, AppJail, and all the other meta-projects (Director, LittleJet, Reproduce, etc.).

I have no intention of running a Linux distro or even Windows, or even another *BSD system other than FreeBSD. This is intentional because I use FreeBSD whenever I can. However, note that the scripts are really customizable or easy to change, so if you are an adventurous user, you can fork this project and modify it to suit your needs.

Requirements

Software

Hardware

This really depends on your needs because the minimum template uses 256 MiB of memory and 20 GiB of storage but this can be changed of course, and probably these resources will not fit to run the application you want, so I assume you know what you really require.

Another requirement is the server or PC to run the WireGuard server using wireguard-xcaler.

I recommend you to use an SSD (or something better) instead of an HDD to get the best performance when creating a virtual machine. You can use a HDD, the problem is when you disable write caching. Enabling write caching increases performance, of course, but at the cost of data integrity.

Installation

I assume you have installed and configured vm-bhyve correctly, but if not, read the vm-bhyve README and wiki.

mkdir -p /cloud-machine
git clone https://github.com/DtxdF/cloud-machine.git /cloud-machine/scripts
cd /cloud-machine/scripts
virtualenv env
cp templates/*.conf /cloud-machine/vm/.templates
. env/bin/activate
pip install -r requirements.txt
mkdir -p components
fetch -o components/MANIFEST 'https://download.freebsd.org/releases/amd64/14.1-RELEASE/MANIFEST'
fetch -o components/base.txz 'https://download.freebsd.org/releases/amd64/14.1-RELEASE/base.txz'
fetch -o components/kernel.txz 'https://download.freebsd.org/releases/amd64/14.1-RELEASE/kernel.txz'
mkdir -p cluster/logs
echo "tmpfs /cloud-machine/scripts/cluster/logs tmpfs rw,late 0 0" | tee -a /etc/fstab
mount /cloud-machine/scripts/cluster/logs

Note: You can use a different version other than 14.1-RELEASE.

Configuration

WARNING: Don't blindly copy files, open your favorite editor and move things around a little at a time.

cp host/dhcpd.conf /usr/local/etc/dhcpd.conf
cp host/loader.conf /boot/loader.conf
cp host/pf.conf /etc/pf.conf
cp host/rc.conf /etc/rc.conf
cp host/sysctl.conf /etc/sysctl.conf
cp host/supervisord.conf /usr/local/etc/supervisord.conf
mkdir -p /usr/local/etc/supervisord.d
cp host/supervisord.d/cloud-machine.ini /usr/local/etc/supervisord.d

Note: Note that these files are tailored to my system and are not portable. For example, my network card using the if_em(4) driver may not make sense in your system.

Files