Skip to content

Key Rotation and Secret Lifecycle

Managed host-Port lifecycle limits

Broker CA rotation on host Port has no supported runbook yet. A rehearsal retired the old CA using hand-built enrollment and material; it does not establish a supported customer procedure. Webhook TLS rotation and provider JWT signing-key refresh do not establish that capability. On the measured Linux host Port + customer relay path, issued store tokens can outlive a deleted Pod until TTL, up to 15 minutes; product revocation attempts were denied. The running test app continued serving after its original tokens expired, with no renewal observed. The direct-auth token renewal description below is a separate contract. Use the customer procedure for the measured limits and explicit restart-based static-KV rotation.

This guide covers rotation procedures for all cryptographic material and credentials used by cloudtaser. Regular key rotation is a requirement for PCI DSS 3.5.1, DORA, and general defense-in-depth.

Leadership summary

Token renewal and rotation depend on the authentication mode and deployment. The managed host-Port limits above apply to the customer relay path. The Transit KEK and direct-auth procedures below have separate prerequisites; they do not establish managed S3 proxy support or immediate Pod-token revocation.

OpenBao Transit KEK Rotation

The cloudtaser S3 proxy uses OpenBao Transit engine for envelope encryption. Each S3 object is encrypted with a unique data encryption key (DEK), which is wrapped by a key encryption key (KEK) managed in OpenBao Transit. The KEK never leaves the EU-hosted OpenBao.

Rotate the KEK

vault write -f transit/keys/cloudtaser/rotate

After rotation:

  • New encryptions use the latest key version automatically
  • Old data remains decryptable -- OpenBao retains all previous key versions by default
  • No re-encryption of existing S3 objects is required (the wrapped DEK metadata in each object references the key version used)

Set Minimum Decryption Version

To enforce that old key versions can no longer decrypt data (after re-encrypting all objects with the new key), set the minimum decryption version:

vault write transit/keys/cloudtaser/config min_decryption_version=N

Where N is the oldest key version you want to keep active. Any wrapped DEK using a version below N will fail to unwrap, making the corresponding S3 objects permanently unreadable unless the minimum version is lowered again.

Verify before advancing min_decryption_version

After rotation, monitor for decryption failures in application logs before advancing min_decryption_version. Advancing prematurely will make objects encrypted with old key versions permanently unreadable.

  • Quarterly rotation of the Transit KEK (minimum)
  • After rotation, monitor for decryption failures before advancing min_decryption_version
  • For regulated workloads (PCI DSS, DORA), document each rotation with date, operator, and key version number

OpenBao Token Lifecycle

cloudtaser components authenticate to OpenBao using Kubernetes auth and manage tokens with different lifecycles.

Wrapper Token Renewal

The wrapper runs as PID 1 in each injected pod. It authenticates to OpenBao on startup using the pod's Kubernetes service account token and receives an OpenBao token. The wrapper automatically renews this token before it expires using periodic renewal. If OpenBao refuses the renewal or it keeps failing, the wrapper treats the token as revoked and terminates itself and the application (fail-closed). The rotation strategy annotation does not apply to token renewal.

Operator Child Tokens

The operator creates short-lived child tokens for specific operations (e.g., token review during Kubernetes auth configuration). These tokens inherit the parent token's policies and have a short TTL.

Token TTL Configuration

Configure token TTLs in the OpenBao Kubernetes auth role:

vault write auth/kubernetes/role/cloudtaser \
  bound_service_account_names=default \
  bound_service_account_namespaces=cloudtaser-system \
  policies=cloudtaser \
  ttl=1h \
  max_ttl=24h
Parameter Description Recommended
ttl Initial token lifetime; the wrapper renews before expiry 1h
max_ttl Absolute maximum lifetime; after this, the wrapper must re-authenticate 24h

Short TTL, frequent renewal

A short ttl (1 hour) with automatic renewal limits the window of exposure if a token is compromised. The max_ttl (24 hours) forces daily re-authentication via Kubernetes auth, ensuring that revoked ServiceAccounts lose access within a day.

Operator TLS Certificate Rotation

The single-operator startup path generates or loads webhook TLS material according to the backend:

Backend Webhook certificate home (inferred from startup code)
inmemory Process memory; a fresh CA and leaf are generated on startup.
vault Vault/OpenBao certificate store, loaded into process memory; a store initialization failure serves a temporary denial listener while initialization retries.

The public CA bundle is patched into the admission webhook configurations. The deprecated kubernetes backend name is normalized to inmemory; it does not select Kubernetes Secret storage. These are code-derived descriptions, not a live rotation qualification. HA has a separate admission trust lifecycle.

Certificate Validity

Certificate or trigger Duration (inferred from operator constants)
CA certificate 730 days
Server/client leaf certificate 90 days
Rotation threshold before expiry 30 days

Automatic Rotation

Inferred from the operator startup wiring: the vault backend checks certificate expiry every 24 hours when its certificate store is initialized, and hot-reloads renewed serving material. The checker is disabled for inmemory; that startup path regenerates ephemeral certificates on restart. This describes the single-operator path, not a measured continuous-rotation or HA guarantee.

See Upgrade & Rollback for backend-specific maintenance limits.

Recovery for the in-memory backend

A restart regenerates the single operator's ephemeral webhook CA and leaf, then republishes public admission trust. Restart before 90 days from operator start, before the leaf expires. After expiry, admission in enrolled namespaces fails closed. Schedule maintenance early enough to restore service: broker and introduction state also lives in RAM. Restore the approved Port introduction and verify readiness and application delivery after the restart. See the host-Port procedure.

kubectl rollout restart deployment/cloudtaser-operator -n cloudtaser-system
kubectl rollout status deployment/cloudtaser-operator -n cloudtaser-system

For the direct vault backend, a restart loads persisted material. It does not force a new CA or revoke a compromised one. Certificate compromise needs a reviewed backend-specific re-issue and trust-replacement procedure; this page does not establish one.

Emergency Procedures

Emergency procedures are disruptive

The following procedures are for incident response only. They will disrupt running workloads. Use them only when a key or credential is compromised.

Volume Shield DEK Revocation

Volume Shield uses one versioned DEK set per encrypted PVC. Destroying every DEK version still referenced by that PVC is a crypto-shredding operation for the PVC's ciphertext, including provider-held snapshots and backups. Use the dedicated Volume Shield DEK revocation runbook for the full procedure, verification steps, and GDPR Article 17 evidence mapping.

OpenBao Token Revocation

Revoke a specific token (e.g., a compromised wrapper token):

# Revoke by accessor (does not require the token itself)
vault token revoke -accessor <accessor>

# Revoke a token directly
vault token revoke <token>

# Self-revoke (from the compromised context)
vault token revoke -self

After revocation:

  • The affected wrapper immediately loses access to OpenBao
  • The wrapper will log authentication errors and (depending on configuration) the pod may restart
  • Other pods and tokens are unaffected
  • The pod will re-authenticate via Kubernetes auth on restart

Transit Key Deletion (Irreversible)

This action is irreversible

Deleting a Transit key permanently destroys all key versions. All data encrypted with this key becomes permanently unrecoverable.

# First, allow deletion (disabled by default)
vault write transit/keys/cloudtaser/config deletion_allowed=true

# Delete the key (IRREVERSIBLE)
vault delete transit/keys/cloudtaser

After deletion:

  • All S3 objects encrypted with this key are permanently unreadable
  • New encryption operations will fail until a new key is created
  • Create a new key with the same name to resume operations: vault write -f transit/keys/cloudtaser

Only use this as a last resort when you need to guarantee that encrypted data can never be decrypted (e.g., data breach containment where the attacker may have copies of encrypted objects and wrapped DEKs).

Operator Certificate Regeneration

Use the backend-specific recovery guidance above. For inmemory, regenerating webhook certificates requires restarting the operator and restoring its RAM state. For vault, restarting alone does not replace persisted compromised material. Verify admission trust and application delivery after recovery.

Full Emergency Lockdown

In a severe compromise, revoke all cloudtaser access:

# 1. Revoke all tokens issued by the Kubernetes auth role
vault token revoke -mode=path auth/kubernetes

# 2. Disable the Kubernetes auth method entirely
vault auth disable kubernetes

# 3. Follow the backend-specific operator certificate recovery above.
#    A restart alone does not replace persisted Vault certificates.

After lockdown:

  • All wrappers lose OpenBao access immediately
  • No new pods can authenticate
  • Applications continue running but cannot fetch new secrets or renew leases
  • Re-enable Kubernetes auth and reconfigure roles to restore access

Applications continue running during lockdown

Existing processes retain their in-memory secrets. They will continue to function until they need to fetch new secrets or renew leases. A lockdown does not terminate running applications.

Multi-Tenant Emergency Lockdown

In a multi-tenant deployment, each cluster registers its own Kubernetes auth mount at auth/kubernetes/<fingerprint>, where <fingerprint> is the kube-system namespace UID of that cluster. This allows you to isolate and revoke access for a single compromised cluster without disrupting other tenants.

Never use the unscoped path in multi-tenant environments

The command vault token revoke -mode=path auth/kubernetes (without a fingerprint suffix) revokes tokens for every cluster's auth mount. In a multi-tenant deployment this is the nuclear option -- it kills all clusters, not just the compromised one. For most incidents, use the scoped revocation below.

Identify the compromised cluster's auth mount

# List all CloudTaser per-cluster auth mounts
vault auth list | grep kubernetes/

# Find the mount for a specific cluster fingerprint
vault read auth/kubernetes/<fingerprint>/config

# If you don't know the fingerprint, get it from the cluster
kubectl get ns kube-system -o jsonpath='{.metadata.uid}'

Scoped revocation (single cluster)

# 1. Revoke all tokens issued by this cluster's auth mount
vault token revoke -mode=path auth/kubernetes/<fingerprint>

# 2. Disable the auth mount to prevent new logins
vault auth disable kubernetes/<fingerprint>

# 3. Follow backend-specific certificate recovery on the affected cluster.
#    Restore approved introduction after an in-memory restart.

After scoped revocation:

  • All wrappers on the targeted cluster lose OpenBao access immediately
  • No new pods on that cluster can authenticate
  • All other clusters continue operating normally -- their auth mounts and tokens are untouched
  • Applications on the targeted cluster retain their in-memory secrets until they need to fetch new secrets or renew leases

Re-enrollment after scoped lockdown

Once the incident is resolved, re-enroll the cluster:

# Option 1: Re-enable the auth mount and reconfigure
vault auth enable -path=kubernetes/<fingerprint> kubernetes
vault write auth/kubernetes/<fingerprint>/config \
  kubernetes_host="https://<cluster-api-server>" \
  kubernetes_ca_cert=@/path/to/cluster-ca.pem

# Option 2: Re-run target install to re-register the cluster
cloudtaser target install --cluster-id <fingerprint> ...

Prefer target install for re-enrollment

Running cloudtaser target install re-registers the cluster, configures the auth mount, applies policies, and deploys the operator in a single command. Manual re-enrollment (option 1) is available when you need fine-grained control over the recovery sequence.

When to use full vs. scoped lockdown

Scenario Action
Single cluster compromised Scoped revocation: vault token revoke -mode=path auth/kubernetes/<fingerprint>
Credential leak affecting one tenant Scoped revocation + rotate the tenant's secrets
OpenBao server compromise Full lockdown: revoke all auth paths, rotate root token, re-enroll all clusters
Uncertainty about which cluster is compromised Full lockdown until the scope of compromise is determined