Introduction
Managed Kubernetes services from major cloud providers can be expensive, with costs quickly escalating as you scale. For developers and small teams, these costs can be a significant barrier. This post details my journey to build a cost-effective, automated Kubernetes as a Service (KaaS) platform on Hetzner Cloud using GitLab CI, Terragrunt, and KubeOne.
Hetzner is known for its affordable and powerful cloud servers, making it an ideal choice for this project. By leveraging open-source tools, I was able to create a robust system that automates cluster creation, scaling, and management, all while keeping costs to a minimum.
The Goal
The primary objective was to create a fully automated system that could provision and manage Kubernetes clusters on demand, with the following key features:
- Automated Provisioning: Use GitLab CI and Terragrunt to create and destroy Kubernetes clusters with a single pipeline run.
- Cost-Effectiveness: Keep infrastructure costs significantly lower than managed Kubernetes offerings.
- Dynamic Scaling: Integrate with a workload manager to allow for autoscaling of VMs based on custom resource definitions.
- Full-Featured Networking: CNI driver and LoadBalancer support for production-ready deployments.
The Tech Stack
This setup relies on a combination of powerful open-source tools to achieve a high degree of automation and control:
- Hetzner Cloud: The underlying IaaS provider, offering cheap and reliable virtual machines.
- KubeOne: An open-source Kubernetes cluster lifecycle management tool that automates the creation and management of clusters.
- Terraform & Terragrunt: Used for defining the infrastructure as code. Terragrunt provides a thin wrapper around Terraform to manage remote state and configurations for different environments.
- GitLab CI: The CI/CD platform used to orchestrate the entire process, from infrastructure provisioning to cluster teardown.
How It Works: The GitLab CI Pipeline
The core of this KaaS platform is a GitLab CI pipeline that automates every step of the process. Here’s a breakdown of the stages:
- 1. Plan: The pipeline starts with a `terragrunt plan` to show the changes that will be applied to the infrastructure. This is a manual step to allow for a final review.
- 2. Apply: Once the plan is approved, a `terragrunt apply` is executed. This provisions the VMs on Hetzner Cloud and runs KubeOne to install and configure the Kubernetes cluster.
- 3. Deploy: With the cluster up and running, you can add jobs to deploy your applications, operators, and other resources.
- 4. Destroy: A manual `terragrunt destroy` job allows you to tear down the entire environment, deleting all resources and stopping the costs. This is perfect for ephemeral development or testing environments.
Cost Calculation: Hetzner vs. The Giants
The main driver for this project was cost. Let's compare a similar setup on Hetzner Cloud versus a managed Kubernetes service like GKE or EKS.
A typical small-scale production setup might include 3 master nodes and 3 worker nodes.
On Hetzner: Using CX21 servers (2 vCPU, 4 GB RAM), the monthly cost for 6 servers would be around €45. Add a load balancer for about €15, and the total is roughly €60/month.
On a major cloud provider: A similar setup with managed Kubernetes can easily cost €200-€300/month or more, including the cluster management fee and higher per-node costs. The savings are substantial.
Advanced Features: Autoscaling with Custom Resources
To make this a true 'as a Service' platform, I implemented autoscaling. This was achieved by creating a custom workload manager that interacts with the Kubernetes API.
A Custom Resource Definition (CRD) was created to define a `VirtualMachine` resource. A custom controller watches for these resources. When a new `VirtualMachine` is created in Kubernetes, the controller calls the Hetzner API to provision a new server and uses KubeOne to join it to the cluster.
This allows for cluster-level autoscaling. For example, a Kubernetes Horizontal Pod Autoscaler can trigger the creation of a new `VirtualMachine` resource when more capacity is needed, which in turn provisions a new physical node.
The Result
The result is a powerful, flexible, and incredibly cost-effective Kubernetes platform. I can spin up a new production-ready cluster for a project in minutes, and tear it down just as easily, paying only for the hours it was running.
The integration with GitLab CI provides a seamless workflow for developers, and the custom autoscaling solution ensures that resources are used efficiently.
This project is a testament to the power of open-source tools and affordable cloud infrastructure. If you're willing to invest some time in setting it up, you can build your own KaaS platform that rivals the functionality of expensive managed services.