Best practices for creating secure, least-privilege controllers with kube.
When we are deploying a
Pod into a cluster with elevated controller credentials, we are creating an attractive escalation target for attackers. Because of the numerous attack paths on pods and clusters that exists, we should be extra vigilant and following the least-privilege principle.
While we can reap some security benefits from the Rust language itself (e.g. memory safety, race condition protection), this alone is insufficient.
Potential Consequences of a Breach#
If an attacker can compromise your pod, or in some other ways piggy-back on a controller's access, the consequences could be severe.
The incident scenarios usually vary based on what access attackers acquire:
- cluster wide secret access ⇒ secret oracle for attackers / data exfiltration
- cluster wide write access to common objects ⇒ denial of service attacks / exfiltration
- external access ⇒ access exfiltration
- pod creation access ⇒ bitcoin miner installation
- host/privileged access ⇒ secret data exfiltration/app installation
See Trampoline Pods: Node to Admin PrivEsc Built Into Popular K8s Platorms as an example of how these types of attacks can work.
Depending on the scope of what your controller is in charge of, you should review and constrict:
|Access Scope||Access to review|
|External||Token permissions / IAM roles|
Managing the RBAC rules requires a declaration somewhere (usually in your yaml/chart) of your controllers access intentions.
Kubernetes manifests with such rules can be kept up-to-date via testing#end-to-end-tests in terms of sufficiency, but one should also document the intent of your controller so that excessive permissions are not just "assumed to be needed" down the road.
RBAC Rules Generation
It is possible to generate rbac rules using audit2rbac (see controller-rs example). This approach has limitations: it needs a full e2e setup with an initial rbac config, and the output may need yaml conversion and refinement steps. However, you can use it to sanity check that your rbac rules are not scoped too broadly.
Installing a CRD into a cluster requires write access to
customresourcedefinitions. This can be requested for the controller, but because this is such a heavy access requirement that is only really needed at the install/upgrade time, it is often handled separately. This also means that a controller often assumes the CRD is installed when running (and panicking if not).
If you do need CRD write access, consider scoping this to non-delete access, and only for the
resourceNames you expect:
- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: NAME rules: - apiGroups: - apiextensions.k8s.io resourceNames: - mycrd.kube.rs resources: - customresourcedefinitions verbs: - create - get - list - patch
Role vs. ClusterRole#
Role (access for a single namespace only) over
ClusterRole unless absolutely necessary.
Some common access downgrade paths:
- if a controller is only working on an enumerable list of namespaces, create a
Rolewith the access
rules, and a
RoleBindingfor each namespace
- if a controller is always generating its dependent resources in a single namespace, you could expect the crd to also be installed in that same namespace.
Deploy the controller to its own namespace to ensure leaked access tokens cannot be used on anything but the controller itself.
The installation namespace can also easily be separated from the controlled namespace.
Follow the standard guidelines for securing your controller pods.
The following properties are recommended security context flags to constrain access:
But they might not be compatible with your current container setup. See documentation of Kubernetes Security Context Object.
For cluster operators, the Pod Security Standards are also beneficial.
Minimizing the attack surface and amount extraneous code in your base image is also beneficial. It's worth reconsidering and finding alternatives for:
debian(out of date deps hitting security scanners)
alpinefor your shell/debug access (escalation attack surface)
scratch(basically a blank default root user)
Instead, consider these security optimized base images:
- distroless base images (e.g.
:ccfor glibc /
- chainguard base images (e.g. gcc-glibc / static for musl)
Supply Chain Security#
If malicious code gets injected into your controller through dependencies, you can still get breached even when following all the above.
Thankfully, you will also most likely hear about it quickly from your security scanners, so make sure to use one.
We recommend the following selection of tools that play well with the Rust ecosystem:
- dependabot or renovate for automatic dependency updates
cargo auditagainst rustsec
cargo auditableembedding an SBOM for trivy /
cargo audit/ syft