Homelab

What it does

My homelab is a three-node Kubernetes cluster that runs everything I self-host - media, ebooks, passwords, and automations - alongside local LLMs and the on-prem half of Security-Copilot. It mixes architectures: an x86 mini PC as the control plane and two arm64 Raspberry Pi 5s as workers, for 16 cores and 96 GB of RAM, with persistent storage on a Synology NAS over NFS.

It is managed with GitOps - every app is a manifest in Git, and Argo CD keeps the cluster in sync with it. It is also hybrid: workloads here share a container image, a cost budget, and data with AWS, over connections that always start from inside the lab.

Why I built it

• Hands-on Kubernetes. Courses and certifications only go so far. I wanted Kubernetes on real hardware, where storage, networking, upgrades, and mixed architectures are problems I actually have to solve.
• Owning my own stuff. I would rather self-host my media, ebooks, passwords, and automations than rent them from SaaS providers.
• A home for my ML projects. An always-on place to run local LLMs, experiment tracking, and my own projects without a cloud bill.

Hardware

Architecture

Git is the source of truth. Argo CD syncs every app into the cluster from a single root application, and Woodpecker CI builds images and pushes them to a private registry on the NAS, tagged with the commit they were built from. Cilium handles pod networking, MetalLB hands out LAN addresses for load-balanced services, and ingress-nginx routes each app's hostname to the right service. Stateful apps get volumes from the NAS through an NFS provisioner, with local-path volumes for node-local data.

  Git repo --> Argo CD                      Woodpecker CI --> registry
                  | syncs every app                        (on the NAS)
                  |                                              | images
+-----------------v----- Kubernetes cluster: kubeadm + Cilium ---v-------+
|                                                                        |
|   webhost1                 webhost5                webhost6            |
|   x86 control plane        arm64 worker            arm64 worker        |
|                                                                        |
|   LAN --> MetalLB --> ingress-nginx --> *.jaredkeyes.site (LAN only)   |
|                                                                        |
+-------------+-------------------------------------------+--------------+
              | NFS                                       | outbound only
              v                                           v
       Synology DS124                             AWS (Security-Copilot)
       app volumes, media,                        ECR        shared image
       backups, registry                          DynamoDB   token budget
                                                  S3         drift data

Hybrid cloud

Security-Copilot runs in two places at once: as a scale-to-zero AWS Lambda and as a deployment on this cluster. The lab connects to AWS only where it has a reason to:

• One image, two platforms. The cluster pulls the exact image digest from ECR that the Lambda runs - one build, two runtimes.
• Shared cost cap. Cloud and on-prem requests draw down the same DynamoDB daily token budget, so one kill-switch covers both.
• Scoped cloud access. The cluster authenticates as a dedicated least-privilege IAM user. Its access key was created outside Terraform, so it never lands in state, and it lives in my self-hosted Vaultwarden.
• Cloud data, on-prem compute. A daily drift-detection job on the cluster reads the report data served from S3 and publishes its results back for the public demo.
• Outbound only. Every AWS integration starts from inside the lab; nothing in the cloud connects in.

The cloud side is covered in detail on the Security-Copilot page.

What runs on it

Platform

• Kubernetes (kubeadm) with Cilium networking
• MetalLB and ingress-nginx for LAN load balancing and routing
• Argo CD app-of-apps managing 14 applications from Git
• Woodpecker CI building images into a private registry on the NAS
• NFS (default) and local-path storage provisioners
• Prometheus, Grafana, and Alertmanager, with node metrics from every node

Machine learning

• Ollama for running LLMs locally
• MLflow tracking server for Security-Copilot, fed by a scheduled job that imports CI evaluation runs from GitHub Actions
• Security-Copilot API - autoscales from 1 to 3 replicas on CPU, plus a daily drift-detection job

Security

• cert-manager with a private homelab CA and Let's Encrypt via DNS-01
• Vaultwarden, a self-hosted Bitwarden-compatible password manager
• App hostnames that resolve to a private LAN address, not a public one
• Nightly backup jobs for Vaultwarden and n8n's Postgres database to the NAS
• A scoped IAM user and outbound-only connections for the AWS side
• Declarative config in Git, so every change is versioned and reversible

Self-hosted apps

• Jellyfin media server
• Calibre-Web Automated, serving an ebook library stored on the NAS
• n8n workflow automation, backed by Postgres

Lessons learned

• NFS permissions need a plan up front: containers run as UID 1000, but Synology users start around 1026, so moving my Calibre library onto the NAS meant sorting out ownership before anything could write to it.
• Mixed architectures shape scheduling: every image has to be multi-arch or pinned to a node, and the heaviest workloads (Ollama, Jellyfin) are pinned to the x86 control plane with a toleration for its taint.
• Keep example files out of applied directories: kubectl apply -f on a folder also applied a secret.example.yaml and overwrote a real Secret with placeholder values.
• Hybrid means expiring credentials: ECR pull tokens only last 12 hours, so an image update on the cluster can fail with ImagePullBackOff long after the last successful deploy.