Introduction

This is a larger guide to showcase how to build controllers, and is a WIP with a progress issue.

Overview

A controller is a long-running program that ensures the Kubernetes state of an object matches the state of the world.

As users update the desired state, the controller sees the change and schedules a reconciliation, which will update the state of the world:

flowchart TD
    A[User/CD] -- kubectl apply object.yaml --> K[Kubernetes Api]
    C[Controller] -- watch objects --> K
    C -- schedule object --> R[Reconciler]
    R -- result --> C
    R -- update state --> K

Any unsuccessful reconciliations are retried or requeued, so a controller should eventually apply the desired state to the world.

Writing a controller requires three pieces:

• an object dictating what the world should see
• an reconciler function that ensures the state of one object is applied to the world
• an application living in Kubernetes watching the object and related objects

The Object

The main object is the source of truth for what the world should be like, and it takes the form of a Kubernetes object like a:

• Pod
• Deployment
• ..any native Kubernetes Resource
• a partially typed or dynamically typed Kubernetes Resource
• an object from api discovery
• a Custom Resource

Kubernetes already has a core controller manager for the core native objects, so the most common use-case for controller writing is a Custom Resource, but other more fine-grained use-cases exist.

Check if your use case fits

Not all use-cases are well-served by a custom controller. Kubernetes.io has a checklist to consider before creating a CRD + a controller for it.

See the object document for how to use the various types.

The Reconciler

The reconconciler is the part of the controller that ensures the world is up to date.

It takes the form of an async fn taking the object along with some context, and performs the alignment between the state of world and the object.

In its simplest form, this is what a reconciler (that does nothing) looks like:

async fn reconcile(object: Arc<MyObject>, data: Arc<Data>) -> Result<Action, Error> {
    // TODO: logic here
    Ok(Action::requeue(Duration::from_secs(3600 / 2)))
}

As a controller writer, your job is to complete the logic that align the world with what is inside the object.
The core reconciler must at minimum contain mutating api calls to what your object is meant to manage, and in some situations, handle annotations management for ownership or garbage collection.

Writing a good idempotent reconciler is the most difficult part of the whole affair, and its difficulty is the reason we generally provide diagnostics and observability.

See the reconciler document for more information.

The Application

The controller application is the part that watches for changes, determines what root object needs reconciliations, and then schedules reconciliations for those changes. It is the glue that turns what you want into something running in Kubernetes.

In this guide; the application is written in rust, using the kube crate as a dependency with the runtime feature, compiled into a container, and deployed in Kubernetes as a Deployment.

The core features inside the application are:

• an encoding of the main object + relevant objects
• an infinite watch loop around relevant objects
• a system that maps object changes to the relevant main object
• an idempotent reconciler acting on a main object

The system must be fault-tolerant, and thus must be able to recover from crashes, downtime, and resuming even having missed messages.

Setting up a blank controller in rust satisfying these constraints is fairly simple, and can be done with minimal boilerplate (no generated files need be inlined in your project).

See the application document for the high-level details.

Controllers and Operators

The terminology between controllers and operators are quite similar:

1. Kubernetes uses the following controller terminology:

In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.

1. The term operator, on the other hand, was originally introduced by CoreOS as:

An Operator is an application-specific controller that extends the Kubernetes API to create, configure and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts, but also includes domain or application-specific knowledge to automate common tasks better managed by computers.

Which is further reworded now under their new agglomerate banner.

They key differences between the two is that operators generally a specific type of controller, sometimes more than one in a single application. To be classified as an operator, a controller would at the very least need to:

• manage custom resource definition(s)
• maintain single app focus

The term operator is a flashier term that makes the common use-case for user-written CRD controllers more understandable. If you have a CRD, you likely want to write a controller for it (otherwise why go through the effort of making a custom resource?).

Guide Focus

Our goal is that with this guide, you will learn how to use and apply the various controller patterns, so that you can avoid scaffolding out a large / complex / underutilized structure.

We will focus on all the patterns as to not betray the versatility of the Kubernetes API, because components found within complex controllers can generally be mixed and matched as you see fit.

We will focus on how the various element composes so you can take advantage of any controller archetypes - operators included.