Skip to content

Zero Kubernetes Secrets Architecture

Identity and store-login boundary

Direct Kubernetes-auth store login authenticates a ServiceAccount and namespace, not the injected wrapper or Deployment. A pod creator using that identity, or node root stealing its JWT, can log in separately and read the role's allowed records. Use one workload per identity, exact-record policies and restricted workload creation. Exec/memory controls do not block that raw login. This limit remains unresolved; see the Security Model. One workload per identity applies on both direct and Port paths. Denial of a second same-identity workload on Port has not been established.

The default operator runtime keeps workload and operator private material in process memory. The chart default is inmemory (inferred from chart values): operational material lives in operator RAM and requires approved introduction after RAM loss. Legacy direct vault mode persists operational material in OpenBao and loads it into memory. Neither backend stores these private credentials in Kubernetes Secrets.

Why

The default runtime aims to keep private material out of etcd; installation inputs have the exceptions listed below. Storing its own operational secrets in Kubernetes Secrets would undermine that promise in several ways:

  • Dogfooding. If cloudtaser protects customer secrets by keeping them out of K8s Secrets, it should do the same for its own. An operator that stores its webhook TLS key in a Kubernetes Secret is one kubectl get secret -o yaml away from compromise.
  • etcd as an attack vector. etcd backups, etcd snapshots, and etcd encryption-at-rest key management are all surfaces where operational secrets could leak. Removing them from etcd eliminates the surface entirely.
  • Unified audit trail. In direct Vault mode, when customer and operational secrets go through OpenBao, every access is recorded in the OpenBao audit log. There is no second audit path to maintain.
  • Sovereignty for operational secrets. cloudtaser's own TLS certificates and broker tokens deserve the same EU-sovereign storage as customer data. In direct Vault mode, the persisted copies can be held in EU OpenBao; serving copies still exist in the operator's process memory.

How Direct Vault Mode Works

Authentication

The operator authenticates to OpenBao using the Kubernetes auth method. It presents its ServiceAccount JWT token to OpenBao, which validates it against the cluster's TokenReview API. No Kubernetes Secret is involved -- the ServiceAccount token is projected into the pod by the kubelet automatically.

Operator Pod
  |
  |  1. Read projected ServiceAccount JWT from /var/run/secrets/kubernetes.io/serviceaccount/token
  |
  v
Vault (K8s auth method)
  |
  |  2. Validate JWT via TokenReview API against the cluster
  |
  v
Return vault token (short-lived, scoped to cloudtaser-operator policy)

First Install

On first startup, when no secrets exist in OpenBao yet, the operator bootstraps itself:

  1. Generates a self-signed CA and webhook TLS certificate
  2. Generates broker mTLS certificates (server, client, CA)
  3. Generates a random broker authentication token
  4. Writes all of these to OpenBao at secret/cloudtaser/system/*
  5. Loads them into memory and begins serving

The default in-memory operator stores runtime private material in process memory. This boundary excludes installation inputs: the legacy chart beacon auth-key input creates a Kubernetes Secret and can enter Helm release state, and the CLI Kubernetes Port bootstrap also creates Secrets. Use the supported host-Port procedure and review the rendered inputs before installation.

Subsequent Starts

On subsequent startups, the operator:

  1. Authenticates to OpenBao via Kubernetes auth
  2. Reads existing secrets from secret/cloudtaser/system/*
  3. Loads them into memory
  4. Begins serving the webhook and broker with the loaded credentials

Webhook TLS

The webhook TLS certificate is served from memory using Go's tls.Config.GetCertificate callback. There is no file mount and no Kubernetes Secret containing the certificate. The API server's MutatingWebhookConfiguration is patched with the CA bundle on every operator startup.

tlsConfig := &tls.Config{
    GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
        return operatorCertManager.GetCurrent(), nil
    },
}

Certificate Rotation

Inferred from the operator startup wiring: the vault backend checks certificate expiry every 24 hours when its certificate store is initialized, and hot-reloads renewed serving material. The checker is disabled for inmemory; that startup path regenerates ephemeral certificates on restart. This describes the single-operator path, not a measured continuous-rotation or HA guarantee.

See the generated validity table and recovery procedure.

High Availability

When the operator runs with multiple replicas (operator.ha: true), all replicas need access to the same broker token so they can authenticate wrapper connections consistently. OpenBao provides this naturally:

  • All replicas read the shared broker token from secret/cloudtaser/system/broker-token
  • Leader election determines which replica actively serves the webhook
  • The standby replicas have the same credentials loaded and can take over instantly

This replaces the pattern of storing shared state in a Kubernetes Secret and watching for changes.

OpenBao Paths

All operator secrets are stored under a single OpenBao path prefix:

Path Contents
secret/cloudtaser/system/webhook-tls Webhook server certificate, private key, and CA certificate
secret/cloudtaser/system/broker-tls Broker mTLS certificates: server cert + key, client cert + key, CA certificate
secret/cloudtaser/system/broker-auth Broker authentication token (used by wrappers to authenticate to the broker)
secret/cloudtaser/system/broker-token Shared broker token for HA replicas (ensures consistent wrapper auth across failover)
secret/cloudtaser/system/enrollment SaaS platform enrollment token (only present when connected to cloudtaser Platform)

All paths use the KV v2 secrets engine. The operator's OpenBao policy grants CRUD access to secret/data/cloudtaser/system/* and secret/metadata/cloudtaser/system/*.

The One Exception

OpenBao bootstrap credentials -- unseal keys and root token -- cannot be stored in OpenBao they unlock. This is an inherent constraint of any OpenBao-based system.

During cloudtaser-cli source install + cloudtaser-cli source configure, these credentials are:

  1. Output once to the terminal
  2. Must be stored externally in an HSM, on paper (Shamir shares), or in an external KMS
  3. Deleted from Kubernetes after retrieval (if they were temporarily stored during automated bootstrap)

Unseal keys are the one secret cloudtaser cannot protect

OpenBao unseal keys and initial root token are the single exception to the zero-K8s-secrets model. Treat them with the highest level of care. Do not store them in the same cloud account as OpenBao. Do not store them in a Kubernetes Secret. The source install command outputs them exactly once.

Default In-Memory Backend

The chart selects the default from its values:

operator:
  secretBackend: inmemory

The operator process uses CLOUDTASER_OPERATOR_SECRET_STORE when configured outside the chart. The in-memory path generates ephemeral operational material at startup and receives store authority through the approved Port/beacon introduction path. It has no direct OpenBao startup dependency. After RAM loss, restore that introduction and verify delivery; persistence and automatic certificate rotation must not be inferred from this default.

Select legacy direct vault mode only for its explicit deployment path and prepare operator store authentication before installation.

Setup for Legacy Direct Vault Mode

The cloudtaser-cli source configure command configures OpenBao for operator secret storage:

cloudtaser-cli source configure \
  --openbao-addr https://vault.eu.example.com \
  --token hvs.YOUR_ADMIN_TOKEN

This creates:

  1. OpenBao policy cloudtaser-operator -- grants CRUD on secret/data/cloudtaser/system/* and secret/metadata/cloudtaser/system/*
  2. Kubernetes auth role cloudtaser-operator -- bound to the operator's ServiceAccount (cloudtaser-operator in namespace cloudtaser-system)

After running this command, deploy or restart the operator with explicit secretBackend: vault. The operator will authenticate, bootstrap its secrets on first run, and serve from memory on subsequent runs.

Preview first

Add --dry-run to see what will be created without making changes: 

cloudtaser-cli source configure --dry-run \
  --openbao-addr https://vault.eu.example.com \
  --token hvs.YOUR_ADMIN_TOKEN

Security Model

The following diagram shows the trust chain with OpenBao-based secret storage. The key property is that no Kubernetes Secret appears anywhere in the chain.

flowchart TD
    subgraph "Kubernetes Cluster"
        SA["ServiceAccount Token<br/>(projected by kubelet)"]
        OP["cloudtaser Operator"]
        WH["Webhook Server<br/>(TLS from memory)"]
        BR["Auth Broker<br/>(mTLS from memory)"]
        API["K8s API Server"]
    end

    subgraph "EU Vault"
        KA["K8s Auth Method"]
        SS["secret/cloudtaser/system/*"]
        AL["Audit Log"]
    end

    SA -->|"1. JWT token"| OP
    OP -->|"2. Login with JWT"| KA
    KA -->|"3. Validate via TokenReview"| API
    KA -->|"4. Return vault token"| OP
    OP -->|"5. Read/write secrets"| SS
    SS -->|"6. Load into memory"| OP
    OP --> WH
    OP --> BR
    SS -.->|"All access logged"| AL

    style SA fill:#e8f5e9,stroke:#2e7d32
    style SS fill:#e3f2fd,stroke:#1565c0
    style AL fill:#fff3e0,stroke:#e65100

What this eliminates:

Traditional approach Zero K8s Secrets approach
Webhook private key stored in a K8s Secret Webhook cert stored in OpenBao, served from memory
Broker token stored in a K8s Secret Broker token stored in OpenBao, loaded into memory
etcd contains cloudtaser TLS private keys Default runtime keeps private material out of etcd; review installation inputs separately
kubectl get secret exposes operator credentials Default runtime creates no operator certificate Secret; installation Secrets can remain
Cert rotation requires Secret update + rollout Direct Vault renewals reload memory; the in-memory backend requires restart
Audit requires K8s audit policy configuration Direct Vault mode: store operations appear in the configured OpenBao audit log

Trust boundaries:

  • The operator trusts the kubelet to project a valid ServiceAccount token
  • The operator trusts OpenBao to authenticate and authorize correctly
  • OpenBao trusts the Kubernetes TokenReview API to validate ServiceAccount tokens
  • No trust is placed in etcd, Kubernetes Secrets, or the cloud provider's control plane