This project started as my first major project with CSH, and has been a work in progress ever since. Due to some pretty significant changes since it's first iteration, documentation for my homelab as of it's first iteration can be found in this repo in the 'old' branch. This main branch documents the newest features and systems running on my homelab.
Since the initial deployment of my homelab, I moved from two servers to one. This change was largely due to the fact that Minerva alone is more than enough to run my entire homelab on, with plenty of resources to spare. Because of this, as well as the issues with relocating and finding a place for a non-rack-mounted server, I sold Eplis, and now my entire homelab resides on Minerva.
- SuperMicro chassis with H8QG6 Motherboard
- RAM - 256Gb DDR3 Registered ECC Memory
- CPU - 48 x AMD Opteron 6180 SE, 4 sockets, 12 cores @2.5GHz
- Networking - 2 x 1Gbe
- Storage
- 2 x 2TiB Hitachi - ZFS Mirror (Boot drive and VM Backups)
- 2 x 2TiB Hitachi - ZFS Mirror (VM Storage)
- 2 x 8TiB WD Gold - ZFS Mirror (Bulk storage)
Many of the basic components of my homelab have remained the same since my first iteration. My basic structure of Proxmox, Pfsense, and Nginx have not changed much since the first iteration of my homelab. Anything in the backend section after Nginx has been added since my first iteration.
- Install Proxmox from the ISO by following the official documentation
- Configure Networking in System>Network
- Once the installation is complete, your system should have a single bridge named vmnbr0 that connects to your primary network interface. This is the interface that will be used for management of proxmox. Add an A level DNS record pointing at this interface's IP address with a name like proxmox.domain.com. Additional services can also be attached to emulated NICs on this bridge, but since I don't want to use up all of the IPs on 50-net all service network traffic will be passed through pfsense.
- Create a new linux bridge and bind it to your secondary network interface. This will be used to pass network traffic into pfsense. Set an A level DNS record pointing at this interface's IP, with the base that you want to use for your services. Format of service URLs will be servicename.thisinterfacename.domain.com.
- Create a third linux bridge and do not bind it to any network interfaces. Set the ip address to 192.168.1.1/24. This is the networking that will be passed through to pfsense
- Configure storage
- Set up your storage in the way you see fit. Due to the drives available to me, I chose ZFS RAID mirrors on Elips and ZFS RAID1 striped RAID on Minerva
- Add certificates in System > Certificates - generate an SSL certificate using something like certbot. Since I already have a wildcard cert for my domain, I just imported it using the proxmox web interface.
- Disable the Enterprise repo and enable the no-subscription repo in the Updates > Repository tab. This will mean you get updates but won't have to pay.
- Installation
- Navigate to local storage > ISO Images. Either download the pfsense ISO from here and re-upload it to proxmox, or use the download from URL feature in proxmox to upload the link generated by the above site.
- Click the 'Create VM' button on the top right of the proxmox dashboard, and create a new VM with 1 socket and 1 Core, and between 512MiB 2GiB RAM. Attach one network device (net0) to the second bridge you created (vmbr1) and a second one to the third bridge you created (vmbr2)
- Start the pfsense VM
- Answer 'n' when the installer asks about VLANs
- Assign WAN to the net0 interface, and LAN to the net1 interface
- Complete the install
- Create a VM with 1 socket, 1 core and 2Gb ram from the Ubuntu Desktop ISO found here. Connect this VM to vmbr2. Complete the ubuntu install process in this VM. Boot this VM and navigate to 192.168.1.1 in your browser. This is where you will manage pfsense. The default username is 'admin' and password is 'pfsense'. Enter these values and configure pfsense.
Routing within my systems is handled by Nginx Proxy Manager. This is a frontend for NGINX that allows for easy remote configuration of proxy hosts, redirection hosts, streams, and 404 hosts. I primarily use NGINX Proxy Manager for it's proxy host functionality, as it routes incoming requests based on my CNAME DNS records to my various service containers. Nginx Proxy Manager also assists with the generation of SSL certificates for some services, and can secure sites behind group-based authentication.
Nginx Proxy Manager has a clean, uncomplicated interface that makes management super easy
This is the proxy-hosts page, where you can see all of my container hosts.
Being able to connect to my server's internal network via a VPN serves two main purposes. First, it makes management of systems like pfsense much easier. Instead of booting a VM inside my network and using VNC through proxmox (SLOW!) to interact with pfsense, with OpenVPN I can just remotely access the management portal as if I were on the same network. Additionally, connecting via OpenVPN enables the management of my k3s cluster from my laptop, which is super convenient and amazing and will be discussed more in the next section of this document. I set up my OpenVPN server using the built-in utility for pfsense, using the instructions here. It was quite a simple setup process, and makes my life so much easier.
In June, I decided to give kubernetes another try, as a way of moving my services out of PCT containers and onto a more compact, scalable platform with lower overhead. I also wanted to learn about kubernetes, and eventually set up some kind of CI/CD, which I got to later (citation needed). Setting up k3s was trivially easy, especially compared to my previous attempts at various types of k8s. I followed this guide until the cluster was all configured, and was accessible by running kubectl get nodes. After that, I worked on moving the majority of my microservices over to k3s, from PCT containers. Initially, I moved heimdall and my personal site over, and started running grafana and archivebox due to the easy configuration on kube. In configuring my system, I set it up so that all cluster configuration would be done from my laptop, while connected to my server's internal LAN, which has greatly simplified the management process. It also makes it much easier to interact with my configuration files, because they are already located locally on my laptop, and doesn't require remotely accessing a control node.
CI/CD
Since the first iteration of my homelab, I noted the services that I tended to use and those that I neglected. I used this information when deciding which services to keep and which to discard in my move to k3s.
Besides Nextcloud (which is special), all of my services run in k3s.
Heimdall - https://dash.vhafener.com
Heimdall is the dashboard I use, which makes it easier and faster to access all of my services. I set this to the default homepage of firefox on all of my machines for easy access.
Nextcloud - https://drive.vhafener.com
This is by far my most used service. This is the primary place where all of my school, personal, and archival documents are hosted. I have a total of 1.6TB currently stored in my Nextcloud account. I use the Nextcloud desktop client to sync my frequently used documents across all of my devices, so that I have local access, while keeping nextcloud up-to-date on my changes. I also configured my Nextcloud instance with a built in Collabora Online Development Environment (CODE) server, which enables real-time sharing and collaboration of documents on my Nextcloud, with very similar performance to google docs. Finally, I have nextcloud configured on my phone to upload photos as I take them, so that I always have the full resolution versions backed up to nextcloud.
Gitlab - https://git.vhafener.com
I use this GitLab instance to hold internal documentation and some repos that I don't want on GitHub, even in private repos. Eventually I plan on integrating this with k8s to handle CD/CI operations.
Overleaf - http://overleaf.vhafener.com
I use overleaf to edit and store my resumes, as well as to typeset important papers or documents
- Optimize Nextcloud with better caching methods
- Implement some kind of SSO
- Proxmox Installation - https://pve.proxmox.com/wiki/Installation
- pfSense Installation - https://docs.netgate.com/pfsense/en/latest/install/download-installer-image.html