Home Lab Experimentation and Learning







In the fast-paced, ever-changing world of information technology, its imperative to keep your skills sharp You've got to constantly challenge yourself to grow and improve and practice your craft. I do this in many ways including self study through online portals such as safari books online, experimentation in a private AWS cloud, and running my own home lab.

The idea to build a home lab is one I picked up from former co-workers whom I really admired in the networking and infrastructure space and ever since then I've been using my own lab to help myself learn new things. This page highlights some of the high level components that make up my home lab and some of the projects I'm pursuing there.

Personality Radar

Virtualization and IaC

The backbone of my homelab is proxmox (mostly because its free and it has an API) I run proxmox on three older hp proliant (1) and dell poweredge server (2) blades I picked up refurbished off amazon for a few hundred bucks. I clustered them together into a single datacenter and slapped some old drives in the raid array and voila, home lab. Its actually evolved over time into this and may continue to evolve further, but this truthd

For infrastructure as code, I use terrraform to provision VMs for all the different 'bare metal' services I run. I use the standard ubuntu cloud image for most things but some microsoft and redhat has crept its way in despite my best efforts. I think I'm up to somewhere around 20 virtual machines in total at the moment. I've got two different HA bare metal vanilla kube clusters. One I use for public facing services like serving this website you're reading right now from my basement, and another private one for my machine learning experiements. I've also got a hashicorp vault server, a gitlab, a dedicated NFS server, some haproxy software 'load balancers', some camera nodes, etc. I also run a few SBCs (single board computers) dedicated for things like pi-hole and my unifi controller.

I use ansible and ansible tower (AWX) to push configuration and I have playbooks for everything from managing machine certs to configuring firewalls, though I by no means consider myself a networking expert. Everything including ssh access is secured with time-bound certificate based authentication managed by my own internal PKI (hashicorp vault) A lot of services run in kube and I go into a bit more detail about that below on as well.

Bare Metal Self Manged On Prem Kubernetes

I have two separate high availablity kubernetes clusters each with three master nodes and 3 worker nodes All together there is around 320GB of memory, 90 CPU cores, and 20 TBs of storage spread across the cluster dedicated to kubernetes. It doesn't do much, but its fun to play with and experiement on.

The load balancers between the master and worker nodes are currently a simple VM running an HA Proxy installation. I eventually want this to be more of a truly HA configuration and use VIPs and Multicast for my load balancer but thats a little beyond my reach at present. My router is set up to forward incoming HTTP traffic requests to my cluster through an ALB set up in a similar fasion, that is also single point of failure in my system, but on my production cluster the ALBs and NLBs are replicated to other nodes via proxmox. Aside from the webpage you're currently reading, I have lots of other payloads running in the cluster but not publicly exposed including things like the following.

  • jenkins
  • prometheus/grafana
  • a secure docker registry
  • postgres and pgadmin
  • mongo and mongo express
  • ansible (awx) tower
  • kubeflow
  • etc

For ingress I run the nginx controller as a daemonset which allows me to route requests very similar to how a real load balancer functions. I also preserve source IPs so I can limit access to certain services to internal IP ranges. I use cert manager to automate issuing certificates and lets-encrypt for public facing services. Internally, I use my own PKI to sign certs and trust my certificate authority on all the machines in my infrastructure using ansible. I use calico as my cluster networking plugin with very little customization. Initially it was a challenge learning and setting all this up even though my setup is pretty basic at its core, but it was super fun and I learned a lot along the way.

The databases are just simple single pod pvc backed implementations. Nothing too ambitious there, all though I have plans in the future to set up an HA postgres cluster with and pgbouncer. I'm always experimenting with my cluster and my next projects include trying gitops with argoCD and rolling out istio or some other service mesh

I'm my own ISP

Sort of...

When I first bought my house, they told me it had high speed internet, but neglected to mention that this was in the form of DSL and to be fair I neglected to check and confirm. This posed a bit of a problem for me initially as I primarly work from home and do a lot of video conferencing and data intensive computer work. I researched internet providers in my area and found a cable provider that offered service nearby, so I contacted them and they came out to see if they could get me a high speed connection.

They quoted me $60,000, which was way out of my price range, and quite frankly ridiculous. They wanted to install a couple of telephone poles, and make a huge deal out of it. However, luckily for me, while the technician was at my house he just happened to mention in passing that there was an existing cable drop already down on another part of my property that was about a quarter of a mile away. (I live in the country and have about 70 acres in the mountains)

I ordered their 1GB service to the existing site, dropped a dry box down there and installed power, a cable modem, and a media converter. Then the next weekend I rented a trencher and spent all day laying two inch PVC conduit from the drop up to my house. I pulled in single mode 12 strand fiber optic cable that I ordered pre-terminated online and in about two weekends for a total cost of around $3,500 including trencher rental I had sutiably fast high speed internet. I typically get around 850-900MB/S down and around 50MB or so up on my wired connections off VPN. That was a pretty dramatic improvement over the unreliable 4MB down and .5MB up I was getting before, and worth every penny :)

Surveillance Cameras

Over time I've installed 16 wired PoE security cameras that run on cat5 to allow me to see the entrance to my property, horse barn, chicken coop, and several interior and exterior views of my home including all entrance doors, approaches, garages etc. I use two DVRs to capture the prior week's worth of video and run periodic backups that dump to an Amazon s3 bucket

I have 4 different TVs dedicated to serving camera feeds over HDMI and I can access my cameras via mobile app from anywhere. I also pipe the camera feeds to a self hosted home assistant instance so I can view them remotely via web browser if I wish.

Networking

My home networking hardware is mostly unifi gear. I have four WAPs (which is massive overkill tbh), a secure network gateway and I host a cloudkey on an SBC using docker. I use it for experimentation, hosting an isolated captive guest wireless network as well as an IoT network and a mixture of wired and wireless devices. Its an experimentation playground and I'm always trying to improve my networking skills and knowledge.

For DNS and DHCP I use pi hole. I do run it on an SBC, although its not a rasberry pi but an x86 based device very similar to a rasberry pi. I configure DHCP reservations for all my servers and create private DNS A records and CNAMEs that point to things like gitlab.ivy.cx for ease of access without setting up an external dns entry to resolve outside my network. Ansible pushes shortcuts to the /etc/hosts file on my bastion host too so I can just type things like ssh master1 to connect via ssh, assuming I first request a signed certicicate from vault.

I use ansible tower to continually push firewall configuration rules to my devices at the host level and at the networking device level, so everything is infrastructure as code with as little UI configuration as possible. As I learn more about networking and ansible I continually expand and add additional functionality. I plan to pursue my CCNA certification this year as well and as I learn more about networking, I'll look to incorporate that into my home lab setup.

Machine Learning

One of the payloads I run in my kube cluster is an image classifier model. The model is a c# port of tensorflow based on the popular inception deep learning nueral network. I trained the model to examine trail cam pictures to detect instances of wildlife. This is part of a polyglot microservice application architecture solution centered around the backend for frontend design methodology. I'm developoing multiple ways to interact with the model including an mvc dotnet core app, a cli client, and a native mobile app designed to deliver notifications, provide analytics and automate a manual task to improve hunting outcomes.

I am a machine learning neophyte and by no means an expert in this field, but I was super proud of myself just for getting a model up and running and doing sort of halfway decent predictions. Before pursuing this route I also experiemented with delivered APIs like Amazon's Rekognition and google's vision API. For quality comparison, my models cannot compete with those services. They are way more accurate, but I am experiementing with how to continually improve my predections via an automated kubeflow pipeline. I have kubeflow up and running in my dev cluster and I'm learning how to leverage it for pipeline automation.

Download Extras

For convenience, You can download my resume, certifications and code samples here.