Skip to content

Security#

Best practices for creating secure, least-privilege controllers with kube.

Problem Statement#

When we are deploying a Pod into a cluster with elevated controller credentials, we are creating an attractive escalation target for attackers. Because of the numerous attack paths on pods and clusters that exists, we should be extra vigilant and following the least-privilege principle.

While we can reap some security benefits from the Rust language itself (e.g. memory safety, race condition protection), this alone is insufficient.

Potential Consequences of a Breach#

If an attacker can compromise your pod, or in some other ways piggy-back on a controller's access, the consequences could be severe.

The incident scenarios usually vary based on what access attackers acquire:

  • cluster wide secret access ⇒ secret oracle for attackers / data exfiltration
  • cluster wide write access to common objects ⇒ denial of service attacks / exfiltration
  • external access ⇒ access exfiltration
  • pod creation access ⇒ bitcoin miner installation
  • host/privileged access ⇒ secret data exfiltration/app installation

See Trampoline Pods: Node to Admin PrivEsc Built Into Popular K8s Platorms as an example of how these types of attacks can work.

Access Constriction#

Depending on the scope of what your controller is in charge of, you should review and constrict:

Access Scope Access to review
Cluster Wide ClusterRole rules
Namespaced Role rules
External Token permissions / IAM roles

RBAC Access#

Managing the RBAC rules requires a declaration somewhere (usually in your yaml/chart) of your controllers access intentions.

Kubernetes manifests with such rules can be kept up-to-date via testing#end-to-end-tests in terms of sufficiency, but one should also document the intent of your controller so that excessive permissions are not just "assumed to be needed" down the road.

RBAC Rules Sanity

It is possible to generate rbac rules using audit2rbac (see controller-rs example). This approach has limitations: it needs a full e2e setup with an initial rbac config, and the output may need yaml conversion and refinement steps. However, you can use it to sanity check that your rbac rules are not scoped too broadly.

See manifests#RBAC for a starter manifest.

CRD Access#

Installing a CRD into a cluster requires write access to customresourcedefinitions. This can be requested for the controller, but because this is such a heavy access requirement that is only really needed at the install/upgrade time, it is often handled separately. This also means that a controller often assumes the CRD is installed when running (and panicking if not).

If you do need CRD write access, consider scoping this to non-delete access, and only for the resourceNames you expect:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: NAME
rules:
- apiGroups:
  - apiextensions.k8s.io
  resourceNames:
  - mycrd.kube.rs # <-- key line
  resources:
  - customresourcedefinitions
  verbs:
  - create
  - get
  - list
  - watch
  - patch

Role vs. ClusterRole#

Use Role (access for a single namespace only) over ClusterRole unless absolutely necessary.

Some common access downgrade paths:

  • if a controller is only working on an enumerable list of namespaces, create a Role with the access rules, and a RoleBinding for each namespace
  • if a controller is always generating its dependent resources in a single namespace, you could expect the crd to also be installed in that same namespace.

Namespace Separation#

Deploy the controller to its own namespace to ensure leaked access tokens cannot be used on anything but the controller itself.

The installation namespace can also easily be separated from the controlled namespace.

Container Permissions#

Follow the standard guidelines for securing your controller pods.
The following properties are recommended security context flags to constrain access:

  • runAsNonRoot: true or runAsUser
  • allowPrivilegeEscalation: false
  • readOnlyRootFilesystem: true
  • capabilities.drop: ["ALL"]

But they might not be compatible with your current container setup. See documentation of Kubernetes Security Context Object.

For cluster operators, the Pod Security Standards are also beneficial.

Base Images#

Minimizing the attack surface and amount extraneous code in your base image is also beneficial. It's worth reconsidering and finding alternatives for:

  • ubuntu or debian (out of date deps hitting security scanners)
  • busybox or alpine for your shell/debug access (escalation attack surface)
  • scratch (basically a blank default root user)

Instead, consider these security optimized base images:

For shell debugging, consider kubectl debug using ephemeral containers instead.

Network Permissions#

Limiting who your controller can talk to / be called by will limit how useful of a target the controller will be in the case of a breach.

It is good practice to setup a default-deny network policy for both ingress and egress and selectively apply as needed.

See manifests#Network Policy for a starter manifest.

Supply Chain Security#

If malicious code gets injected into your controller through dependencies, you can still get breached even when following all the above.
Thankfully, you will also most likely hear about it quickly from your security scanners, so make sure to use one.

We recommend the following selection of tools that play well with the Rust ecosystem:

References#