Upgrade and Rollback Procedures

Upgrade and rollback requirements depend on the source and target cohort.

Current host-Port release: fresh install only

In-place upgrade from older charts to 1.0.359 was not tested on this path. The customer guide covers fresh install only. Do not use the historical procedures below as qualification for this release. Skip intermediate charts 1.0.357 and 1.0.358.

Older broker-path Pods can lose their next secret read with 403 not-registered until recreated with wrapper v0.2.207. A container restart does not replace injected binaries or limits. Before recreation, set token auth, one selected wrapped container and a scoped CloudTaserPolicy with non-empty vaultPaths; default kubernetes auth is denied at CREATE on this broker path. Preserve enforcement and review the matched chart/site inputs before maintenance.

Rollback can require Pod recreation and another CA rotation

A rollback to chart 1.0.355 pairs operator v0.13.205's TLS-bearing payloads with newer wrappers that refuse them (HTTP 400); Pods stay sealed until recreated on that chart's wrapper v0.2.202. The old webhook can also deny metadata updates to Pods created with the newer copy-init command. Recreate Pods on the old wrapper, then treat the broker CA as exposed and rotate again: the older operator re-delivers the broker private key to workloads. Host Port has no supported broker CA rotation runbook in this release, so this is not a supported self-service recovery procedure. Arrange source/cluster maintenance with your administrators before choosing a downgrade. Rolling back can also restore the old default memory cap.

---

Upgrading to chart 1.0.20+ (from 0.4.x or 1.0.0-1.0.19)

Chart 1.0.20 completes the sealed-TLS-from-t0 cutover. Three contracts changed in load-bearing ways; each has a distinct pre-flight check. Read this section in full BEFORE running helm upgrade. The matching in-cluster NOTES.txt block is reproduced in charts/cloudtaser/templates/NOTES.txt in the cloudtaser-helm repo.

A. Audit stored values before helm upgrade --reuse-values

helm upgrade --reuse-values merges the NEW chart defaults on top of your OLD stored values. Chart 1.0.0 introduced operator.broker.*, which did not exist in 0.4.x - so the defaults land cleanly on a fresh release, but on an upgraded release the NEW default operator.broker.tls.enabled=true starts serving TLS and the operator v0.6.30 refuse-to-start guardrail (section D below) will abort helm upgrade if your stored values carried a stale operator.broker.tls.enabled=false from an intermediate pin WITHOUT the matching operator.broker.allowHTTP=true.

The failure surfaces DURING helm upgrade (clear error, no silent drift) - but only after the release has been partially mutated. You want to catch it BEFORE running the command.

Run this first:

helm get values cloudtaser -n cloudtaser-system > /tmp/stored-values.yaml
grep -E '^\s*(broker|tls|allowHTTP|enabled):' /tmp/stored-values.yaml || \
  echo 'no broker overrides stored - safe to upgrade'

If you see operator.broker.tls.enabled: false in the output, add operator.broker.allowHTTP: true to the same values file BEFORE upgrading (or, recommended, remove the override entirely and accept the TLS default). If you see only operator.broker.tls.enabled: true or nothing, you are safe to proceed.

Either way, bump cloudtaser-cli to v0.16.2+ BEFORE the upgrade. The new CLI auto-fetches the broker CA from the cloudtaser-operator-broker-tls Secret and auto-upgrades to https://. Older CLIs dial http:// and fail with tls: first record does not look like a TLS handshake.

B. Two-phase upgrade order - operator-first wedges running pods

Operator v0.6.30 dials https:// for pod unseal unconditionally. It does NOT fall back to plain HTTP - silent fallback would violate the sealed-TLS-from-t0 invariant. Wrapper v0.1.14 serves :8199 as plain HTTP; wrapper v0.1.15 serves :8199 over TLS.

DO NOT bump operator image first

DO NOT bump operator.image.tag to v0.6.30 (or upgrade the chart with operator-image defaults) while ANY running pod still carries a wrapper v0.1.14 binary. The new operator's HTTPS unseal dial fails against the old wrapper's plain HTTP listener, and injected readiness probes use the HTTPS scheme against old-wrapper pods. The failure mode is connection refused / bad record version and looks like a NetworkPolicy problem - it is not. Mixed-version protected pods will crashloop until wrapper is upgraded AND the pod is recreated.

Follow these four steps in order. No skipping.

Step 1. Upgrade the chart with wrapper image bumped, operator pinned:

helm upgrade cloudtaser cloudtaser/cloudtaser --version 1.0.20 \
  --set operator.image.tag=v0.6.29 \
  --set wrapper.image.tag=v0.1.15 \
  --reuse-values

Step 2. Recreate every already-injected pod so the new wrapper binary AND the new webhook-injected HTTPS probes land. Use kubectl rollout restart (not kubectl delete pod) so Deployments/StatefulSets/DaemonSets respect surge settings:

kubectl get deployments,statefulsets,daemonsets --all-namespaces \
  -o jsonpath='{range .items[?(@.spec.template.metadata.annotations.cloudtaser\.io/inject=="true")]}{.kind}/{.metadata.namespace}/{.metadata.name}{"\n"}{end}' | \
  while IFS=/ read kind ns name; do
    kubectl rollout restart "${kind,,}/${name}" -n "${ns}"
  done

Step 3. Verify every injected pod is Ready (nothing stuck in Init or CrashLoopBackOff) BEFORE moving to step 4:

kubectl get pods -A -l cloudtaser.io/protected=true | \
  grep -v '1/1\|2/2\|3/3\|Running\|Completed' || \
  echo 'all injected pods Ready - safe to bump operator'

Step 4. Bump the operator image (now safe - every wrapper is v0.1.15):

helm upgrade cloudtaser cloudtaser/cloudtaser --version 1.0.20 \
  --reuse-values

Do not merge steps 1 and 4

Each step must complete and be verified independently. DO NOT merge steps 1 and 4 into a single helm upgrade unless you have zero already-running protected pods (fresh install, or all protected Deployments scaled to 0). Fresh installs can skip directly to the 1.0.20 defaults - operator v0.6.30 + wrapper v0.1.15 are a matched pair.

C. Prometheus scrape migration - hostPort removed

The ebpf daemonset no longer publishes a hostPort: 9099. Scrapers configured against node:9099/metrics receive zero samples with no error (the node port is simply closed). The agent now splits its HTTP server into two listeners inside the daemonset pod's network namespace:

Listener	Endpoint	Reachable
pod-IP:9099	/healthz, /readyz	Public via cloudtaser-ebpf-agent-metrics Service
127.0.0.1:9098	/metrics, /status	Loopback-only (intentional)

/metrics AND /status are INTENTIONALLY unreachable from off-pod. The cloudtaser-ebpf-agent-metrics Service (which is NOT annotated with prometheus.io/scrape by the chart) fronts pod-IP:9099 only - it cannot reach /metrics on 127.0.0.1:9098 no matter how you configure the ServiceMonitor. See the upstream audit at loopback metrics security audit for the security rationale.

You have three honest options. There is no middle path.

Option A - Scrape /healthz + /readyz as an up-indicator only. The only off-pod observable surface on the Service is the readiness of each daemonset pod. This tells you the agent is alive but gives you NO runtime metrics (block counters, kprobe stats, event rates). Acceptable for platform teams who only need an "is the enforcement layer running everywhere" dashboard; NOT sufficient for detection engineering on block-event counts.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cloudtaser-ebpf-agent-up
  namespace: cloudtaser-system
  labels:
    release: prometheus      # whatever your operator selects on
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudtaser-ebpf
  endpoints:
    - port: health
      path: /healthz
      interval: 30s

The up{} series plus an HTTP 200 on /healthz is all you will get over this Service.

Option B - Deploy an in-pod sidecar that scrapes loopback. Add a prometheus-agent (or vmagent, otel-collector - any scraper) sidecar to the ebpf daemonset with hostNetwork: false and its own network namespace shared with the agent container, then point it at 127.0.0.1:9098/metrics and remote-write to your long-term store. This is the ONLY way to get the full /metrics firehose off-cluster while preserving the loopback-only security property. The chart does not ship a built-in sidecar values block today; track chart-native metrics scraper proposal for the chart-native variant.

Option C - Accept pod-local metrics. If you do not need /metrics aggregated centrally, leave it loopback. An attestation agent running inside the same pod (Falco, custom) can consume 127.0.0.1:9098/metrics and 127.0.0.1:9098/status directly. No chart changes required.

Do not target /metrics on the Service

Do NOT write a ServiceMonitor that targets cloudtaser-ebpf-agent-metrics with path: /metrics or path: /status. The Service only exposes port 9099, which serves /healthz + /readyz only. Prometheus will scrape the endpoint, receive HTTP 404 for /metrics or /status, and drop the series silently - same failure mode as the old node:9099/metrics scrape, just relocated to the Service. Rewrite your scrape config to one of the three options above.

If you already had serviceMonitor.enabled=true in your values, it continues to cover the OPERATOR metrics - you need an additional ServiceMonitor (or equivalent scrape config) for the ebpf agent.

D. operator.broker.tls.enabled=false now requires operator.broker.allowHTTP=true

Operator v0.6.30 adds a refuse-to-start guardrail: if you disable broker TLS you MUST explicitly opt in to plain HTTP. The chart enforces this at helm template time with a clear error; you cannot apply a bricked Deployment. Update your values.yaml override:

operator:
  broker:
    tls:
      enabled: false
    allowHTTP: true          # REQUIRED alongside tls.enabled=false

If you have no such historical override, this section does not apply - the default (tls.enabled=true, allowHTTP=false) needs no change.

---

Section E: May 2026 -- Beacon and proxy TLS hardening

The May 2026 wave hardens four components: beacon relay, CloudTaser Port, S3 proxy, and DB proxy. Most changes are transparent -- secure defaults tighten without requiring operator action. Two changes require explicit attention if you use development-mode TLS bypass flags.

E1: Beacon hardening (beacon v0.3.x+)

Three changes landed in beacon PRs IPv6 rate limiting and map bounds, Loopback drain listener, and TCP keepalive before TLS.

IPv6 rate-limiter fix

The beacon's per-IP rate limiter now aggregates IPv6 addresses to their /64 prefix. Previously, an attacker rotating through addresses within a single /64 block got a fresh rate-limit bucket per address, effectively bypassing the limiter. IPv4 per-IP behaviour is unchanged.

The rate-limit map is also capped at 10,000 entries with oldest-entry eviction, preventing unbounded memory growth under sustained attack from diverse source IPs.

Operator action: none. The fix is automatic.

Admin drain endpoint now loopback-only

The /admin/drain endpoint is now served on a dedicated loopback-only listener when the health server binds to a non-loopback address. Previously, /admin/drain was protected only by a RemoteAddr check, which can be spoofed by a reverse proxy. Now the network layer enforces access: the dedicated listener binds to 127.0.0.1 on the same port as the health server, so only processes running on the beacon host can reach it.

When the health server is already bound to loopback (the default 127.0.0.1:8080), /admin/drain remains on the health mux as before -- the loopback bind already provides the same guarantee.

Operator action: if you call /admin/drain from a remote host (e.g. a load-balancer health check or an external orchestrator), that will no longer work. Move your drain invocation to run on the beacon host itself (e.g. via SSH, a sidecar, or a pre-stop lifecycle hook). Calling drain from outside the beacon host was never recommended -- it exposed an unauthenticated state-mutation endpoint to the network.

TCP keepalive before TLS handshake

TCP keepalive probes are now applied immediately after Accept, before the TLS handshake begins. Previously, keepalive was applied only after a successful handshake, which meant half-open connections that stalled during the TLS handshake could accumulate without being cleaned up by keepalive probes.

Operator action: none. The fix is automatic and prevents half-open connection accumulation under adverse network conditions.

E2: CloudTaser Port vault TLS dual-flag requirement

Paired TLS opt-in adds a paired-flag guard to the port's vault TLS skip-verify behaviour. Previously, setting --vault-skip-verify=true (or VAULT_SKIP_VERIFY=true in the environment) alone was sufficient to disable vault TLS certificate verification. A single stale environment variable copied from a demo .env file could silently disable TLS verification in production.

Now, both --vault-skip-verify=true and --vault-allow-insecure=true must be set. If only --vault-skip-verify is set, the port refuses to start with a clear error:

--vault-skip-verify=true requires --vault-allow-insecure=true

The same guard applies to the environment variable form:

# Old (no longer works alone):
VAULT_SKIP_VERIFY=true

# New (both required):
VAULT_SKIP_VERIFY=true
CLOUDTASER_PORT_VAULT_ALLOW_INSECURE=true

Additionally, the backup/restore scripts and the enroll server now default to https://127.0.0.1:8200 instead of http://127.0.0.1:30200. The admin/root token was previously transmitted in cleartext over loopback.

Operator action for production deployments with proper TLS certificates: none. The flags default to false and proper TLS verification is the default path.

Operator action for development setups using --vault-skip-verify: add --vault-allow-insecure=true alongside --vault-skip-verify=true. This is intentional friction -- two flags must agree before TLS verification is disabled, preventing a single stale env var from degrading security.

E3: S3 proxy security defaults (s3-proxy v0.3.x+)

Proxy hardening hardens three areas:

Dual-flag vault TLS guard

Same pattern as the port (E2 above). CLOUDTASER_S3PROXY_VAULT_SKIP_VERIFY=true now requires CLOUDTASER_S3PROXY_VAULT_ALLOW_INSECURE=true. The proxy refuses to start if only the skip-verify flag is set.

Health and metrics servers bound to loopback

The health server (/healthz, /readyz) default address changed from :8191 to 127.0.0.1:8191. The Prometheus metrics server default changed from :9090 to 127.0.0.1:9090. Both are now loopback-only by default, preventing exposure of health and metrics endpoints to the pod network.

If you need to expose these endpoints to Kubernetes probes or a ServiceMonitor, override the addresses explicitly:

env:
  - name: CLOUDTASER_S3PROXY_HEALTH_ADDR
    value: "0.0.0.0:8191"    # needed if kubelet probes target the pod IP
  - name: CLOUDTASER_S3PROXY_METRICS_ADDR
    value: "0.0.0.0:9090"    # needed if Prometheus scrapes the pod IP

Kubelet probes may need updating

If your liveness/readiness probes target the pod IP, they cannot reach a loopback-bound health server. For a wrapped/protected S3 proxy, bind CLOUDTASER_S3PROXY_HEALTH_ADDR to the Pod IP or 0.0.0.0:8191 and use an httpGet probe; restrict endpoint access with your network policy. An exec probe that curls 127.0.0.1:8191/healthz is an option only for an unprotected container: default runtime enrollment currently denies exec bootstrap in protected containers. See Troubleshooting.

TLS 1.3 minimum for vault connections

All TLS connections from the S3 proxy to vault now enforce TLS 1.3 as the minimum version. Previously, the Go default (TLS 1.0) was used. This closes the window for protocol-downgrade attacks on the proxy-to-vault path.

Operator action: ensure your vault server supports TLS 1.3. All OpenBao versions and HashiCorp Vault 1.12+ support TLS 1.3 by default. If your vault is behind a TLS-terminating proxy, verify that the proxy negotiates TLS 1.3.

E4: DB proxy security defaults (db-proxy v0.2.x+)

Database proxy hardening applies the same three hardening patterns as the S3 proxy, plus a fail-closed encryption guard:

Fail-closed on vault transit outage

The DB proxy now rejects queries when encryption fails (vault unreachable, transit key missing, auth expired), instead of silently forwarding plaintext to the upstream database. This is the fail-closed default (CLOUDTASER_DBPROXY_ENCRYPT_FAIL_MODE=fail-closed).

For mid-migration scenarios where some queries must continue during vault unavailability, set CLOUDTASER_DBPROXY_ENCRYPT_FAIL_MODE=fail-open explicitly. This preserves the old behaviour but must be an intentional opt-in.

fail-open leaks plaintext

fail-open mode means a vault outage silently passes plaintext to the cloud-provider-hosted database. Use it only during migration windows and revert to fail-closed (the default) as soon as migration is complete.

Dual-flag vault TLS guard

Same pattern as E2 and E3. CLOUDTASER_DBPROXY_VAULT_SKIP_VERIFY=true requires CLOUDTASER_DBPROXY_VAULT_ALLOW_INSECURE=true.

Health server bound to loopback

Default health address changed from :9091 to 127.0.0.1:9091. Same considerations as E3 -- override if kubelet probes target the pod IP.

TLS 1.3 minimum for vault and database connections

All TLS connections (vault client, PostgreSQL upstream, MySQL upstream) now enforce TLS 1.3 minimum.

E-series summary

Change	Component	Breaking?	Action required
IPv6 /64 rate-limiter aggregation	Beacon	No	None
Admin drain loopback-only	Beacon	Yes, if called remotely	Move drain calls to beacon host
TCP keepalive before TLS	Beacon	No	None
Vault TLS dual-flag guard	Port (cloudtaser-port)	Yes, if using --vault-skip-verify alone	Add --vault-allow-insecure=true
Vault TLS dual-flag guard	S3 proxy	Yes, if using skip-verify alone	Add CLOUDTASER_S3PROXY_VAULT_ALLOW_INSECURE=true
Vault TLS dual-flag guard	DB proxy	Yes, if using skip-verify alone	Add CLOUDTASER_DBPROXY_VAULT_ALLOW_INSECURE=true
Health/metrics loopback bind	S3 proxy	Yes, if probing via pod IP	Override health/metrics addr to 0.0.0.0
Health loopback bind	DB proxy	Yes, if probing via pod IP	Override health addr to 0.0.0.0
TLS 1.3 minimum	S3 proxy, DB proxy	Possibly, if vault < TLS 1.3	Verify vault supports TLS 1.3
Fail-closed encryption	DB proxy	Yes, if relying on silent plaintext fallback	Set fail-open explicitly during migration, or fix vault connectivity
Backup/restore HTTPS default	Port (cloudtaser-port)	No (loopback only)	None

---

Section F: May 2026 -- Wrapper health server hardening and crash-loop prevention

The wrapper health server received three hardening changes and a crash-loop prevention fix. All changes are backward-compatible with no operator action required for standard deployments.

F1: Health server timeouts and body limits (wrapper v0.2.12+)

The health server now enforces server-side timeouts to prevent slowloris and connection-holding attacks:

Setting	Previous	New
ReadTimeout	Not set (unlimited)	30s
WriteTimeout	Not set (unlimited)	30s
IdleTimeout	Not set (unlimited)	120s
MaxHeaderBytes	Not set (Go default 1MB)	64 KB
Unseal body limit	256 KB	64 KB
Body read deadline	Not set	10s per request

Operator action: none. The limits are generous for legitimate traffic. The 64 KB unseal body limit accommodates the ECDH-encrypted payload (cert chain + key + token + nonce + ciphertext) with room to spare.

F2: Per-source-IP rate limiting on /v1/unseal (wrapper v0.2.12+)

The unseal rate limiter is now per-source-IP (100 req/min per IP, burst of 10) with a global cap (1000 req/min, burst of 50), replacing the previous global-only 10 req/min limiter. This prevents a co-located attacker from exhausting the rate limit budget and locking out the legitimate operator.

Rate limiting is applied after JWT authentication, so unauthenticated requests are rejected before consuming rate limit tokens.

Operator action: none. The new limits are strictly more permissive for legitimate operator traffic (the operator's single IP gets 100 req/min instead of sharing a global budget of 10 req/min).

F3: Secret-not-provisioned stay-alive behaviour (wrapper v0.2.12+)

When the upstream secret has not been provisioned in vault yet, the wrapper now stays alive indefinitely with bounded-backoff retries instead of calling os.Exit(1). This prevents the CrashLoopBackOff cascade where kubelet restart backoff escalates to 5 minutes, making the pod unrecoverable even after the secret is later provisioned.

Health endpoint behaviour during this state:

• /healthz returns 200 (wrapper PID 1 is alive)
• /readyz returns 503 with {"ready":false,"reason":"secret_not_provisioned"}

Only configuration errors (auth failure, invalid vault path) still trigger os.Exit.

Operator action: none. This is strictly better behaviour -- pods that previously crash-looped on missing secrets now self-heal when the secret appears.

F4: RLIMIT_CORE=0 and envmap format injection guard (wrapper v0.2.12+)

Two additional security hardening fixes:

1. RLIMIT_CORE=0 -- The wrapper sets both soft and hard core dump rlimits to zero as defense-in-depth alongside PR_SET_DUMPABLE(0). This blocks core dump generation via systemd-coredump, core_pattern pipes, or container runtimes that override the dumpable bit.

2. Envmap format injection guard -- The CLOUDTASER_ENV_MAP parser now uses explicit type handling instead of Go's %v format verb. String values containing newline, carriage return, or NUL bytes are rejected to prevent environment variable injection through crafted vault values.

Operator action: none. Both are transparent hardening improvements.

F-series summary

Change	Component	Breaking?	Action required
Server-side timeouts (Read/Write/Idle)	Wrapper	No	None
Unseal body limit reduced (256KB to 64KB)	Wrapper	No	None
Per-IP rate limiting on unseal	Wrapper	No (more permissive for legitimate traffic)	None
Secret-not-provisioned stay-alive	Wrapper	No (strictly better)	None
RLIMIT_CORE=0 defense-in-depth	Wrapper	No	None
Envmap format injection guard	Wrapper	Possibly, if vault values contain newline/CR/NUL	Remove injection characters from vault values

---

Section G: May 2026 -- Beacon unauthenticated surface hardening

Four changes harden the beacon's unauthenticated attack surface against scanner noise, gossip-based DoS, and connection exhaustion. All landed in beacon relay surface hardening (2026-05-15) and are included in beacon v0.4.x+.

G1: Scanner log demotion

Ten noisy log lines triggered by port scanners and health probes (TLS handshake failures, unexpected protocol bytes, premature connection close) have been demoted from WARN to DEBUG in internal/beacon/server.go.

Operator action: if your log aggregation pipeline filters on WARN-level beacon entries to detect attacks, those scanner-triggered lines will no longer appear at WARN. Adjust your filters to include DEBUG if you relied on them, or -- preferably -- alert on the rate-limit and concurrency-cap events described below, which are a more reliable signal of sustained abuse.

G2: Gossip size cap

ACME gossip entries (used by the beacon's distributed Let's Encrypt solver) are now capped at 8,192 bytes. Both the Store() write path and the handleUserMsg() receive path reject values exceeding the cap and return errGossipValueTooLarge.

This prevents a gossip-based DoS where a malicious or misconfigured peer floods the cluster with oversized gossip values that consume memory across all beacon replicas.

Operator action: none under normal operation. ACME certificates and challenge tokens are well within 8 KB. If you see errGossipValueTooLarge in beacon logs, a peer is sending malformed or malicious gossip -- investigate the source.

G3: Forward rate limit

Each client IP is limited to 100 forward connections per minute. IPv6 addresses are bucketed by /64 prefix (consistent with the existing rate-limiter aggregation from Section E1). Connections exceeding the limit receive a rateLimit error and are closed.

This caps the blast radius of a single-source connection flood. The 100/min default is generous for legitimate port and broker traffic (a single cluster typically maintains 1-3 persistent connections).

Operator action: none for standard deployments. If you operate a beacon serving a very large number of clusters behind a shared NAT (where many distinct clusters share the same source IP), the 100/min limit may be reached. This is an unlikely topology for production -- each cluster's broker maintains a small number of long-lived connections, not bursts of short-lived ones. If you encounter rateLimit errors in the beacon log for legitimate traffic, contact the cloudtaser team to discuss raising the default.

G4: Concurrency cap

The beacon enforces a maximum of 1,000 simultaneous inner forwarded connections via a buffered channel semaphore in NewForwardServer. When the cap is reached, new connections block (they are not dropped) until a slot frees up. This is a backpressure mechanism, not a hard reject -- legitimate connections that arrive during a spike will proceed once capacity is available.

The cap prevents unbounded goroutine and file-descriptor growth under sustained connection floods, protecting the beacon process from OOM kills and file-descriptor exhaustion.

Operator action: none for standard deployments. A single port-broker pair uses a small number of forwarded connections. If you host a single beacon for hundreds of clusters and observe sustained blocking (visible as elevated connection latency in beacon metrics), increase the beacon replica count or deploy per-region beacon instances.

G-series summary

Change	Issue	Breaking?	Action required
Scanner log demotion (WARN to DEBUG)	scanner log demotion	No (log level only)	Adjust WARN-based log filters if needed
Gossip size cap (8,192 bytes)	gossip size cap	No	None
Forward rate limit (100/min per IP, IPv6 /64)	forward rate limit	Possibly, behind shared NAT with many clusters	None for standard deployments
Concurrency cap (1,000 simultaneous connections)	connection concurrency cap	No (blocks, not drops)	None for standard deployments

---

Section H: May 2026 -- Operator broker auth hardening

Five security fixes in operator PRs broker authentication hardening and broker response and environment hardening harden the broker's HTTP surface. Three of the five changes are breaking for automation that interacts with the broker directly; the other two are transparent improvements.

H1: /v1/port/status now requires Bearer token authentication (operator v0.12.x+)

The /v1/port/status endpoint previously returned port connectivity state to any unauthenticated in-cluster caller. This leaked the beacon address and connection status to any pod that could reach the operator's broker port.

The endpoint now requires a valid Authorization: Bearer <token> header. The token is the same operator auth token used for other authenticated broker endpoints (/v1/port/init, /v1/proxy/admin).

Responses without a valid token:

Condition	HTTP status	Response body
Missing or malformed Authorization header	401	{"error":"unauthorized"}
Invalid bearer token	401	{"error":"unauthorized"}
No auth token configured on the operator (fail-closed)	403	{"error":"broker auth not configured"}

Operator action: if you have automation or monitoring that polls /v1/port/status, add the Authorization: Bearer <token> header. The token is the same one used for /v1/port/init. Example:

curl -s -H "Authorization: Bearer ${OPERATOR_AUTH_TOKEN}" \
  https://cloudtaser-operator.cloudtaser-system:8443/v1/port/status

If you only interact with the operator through the CLI (cloudtaser-cli target status), no action is required -- the CLI handles authentication automatically.

H2: Unseal token format validation (operator v0.12.x+)

The /v1/unseal endpoint now performs a lightweight format check on the vault token before forwarding it to vault. Tokens must be:

• At least 8 characters long
• Composed entirely of printable ASCII characters (0x20-0x7E)

Tokens that fail either check are rejected with HTTP 400 ({"error":"malformed token"}) and never reach vault. Previously, any string -- including empty strings, single characters, or strings containing control characters -- was forwarded to vault, which would reject them with a less informative error.

This is a fail-fast sanity check, not a cryptographic validation. Vault remains the authority for token validity. Real vault/OpenBao tokens (prefixed s., hvs., or raw UUID format) are 26+ characters and always pass this check.

Operator action: none for standard deployments. The operator's unseal flow sends properly formatted tokens. This change only affects tooling that sends hand-crafted requests to /v1/unseal with malformed tokens.

H3: Error response sanitization (operator v0.12.x+)

Vault error responses returned through the broker's CreateChildToken path are now sanitized before being returned to callers. Previously, vault error details -- which can contain accessor IDs, policy names, and cluster topology information -- were forwarded to callers. In a US-cloud deployment, these details could propagate to US-hosted SIEMs and logging pipelines, undermining the data sovereignty guarantee.

The caller now receives a generic error with the HTTP status code only:

vault token creation failed (status 403)

The full vault error is logged at Debug level on the operator side, accessible via kubectl logs for operators who need to diagnose the root cause.

Operator action: if your automation parses vault error messages returned by the broker, update it to handle the generic format. For debugging, check the operator's Debug-level logs:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator \
  --tail=200 | grep "sanitizedError"

H4: JWT scrubbing from response bodies (operator v0.12.x+)

The broker's /v1/proxy/secret handler now scrubs the calling pod's service-account JWT from all response bodies before writing them to the wire. A malicious or misconfigured port could echo the pod's JWT back in an error message; any in-cluster caller that can read the broker's response (the secret proxy does not use bearer-token auth -- it authenticates via the JWT in the request body) would then obtain the pod's service-account token.

Scrubbing replaces any occurrence of the full JWT string with [scrubbed-jwt] in both the structured response and serialized JSON output.

Operator action: none. This is a transparent defense-in-depth improvement. If you see [scrubbed-jwt] in broker responses during debugging, it means the port echoed a JWT that was correctly stripped.

H5: Dynamic-linker environment variable stripping at admission (operator v0.12.x+)

The mutating admission webhook now strips dangerous dynamic-linker environment variables from all containers in injected pods before the pod is created. The following prefixes are removed:

Prefix	Risk
LD_*	Arbitrary shared-library injection via LD_PRELOAD, LD_LIBRARY_PATH
GLIBC_TUNABLES	glibc tunable abuse (CVE-2023-4911)
GCONV_PATH	Character-set converter path hijack
MALLOC_*	Heap allocator configuration manipulation
NSS_*	Name Service Switch module hijack
OPENSSL_*	OpenSSL configuration override

Previously, the wrapper stripped these from its own child's environment at fork+exec time. However, kubectl exec-spawned shells inherit the pod-spec environment directly and never pass through the wrapper -- so a hostile LD_PRELOAD value in the pod spec persisted into debug shells. Stripping at admission closes both vectors.

The cloudtaser operator's own LD_PRELOAD=/cloudtaser/libcloudtaser.so (the getenv interposer) is not affected -- dangerous env vars are stripped from the user's container spec before the operator injects its own environment variables.

Existing pod specs with legitimate loader env vars

If your container specs set LD_LIBRARY_PATH for legitimate reasons (e.g. custom shared-library paths for native dependencies), those values will be silently stripped after this upgrade. Move custom library paths into the container image's /etc/ld.so.conf.d/ configuration or bake them into the LD_LIBRARY_PATH inside the Dockerfile's ENV layer (which the webhook does not modify -- only pod-spec env entries are filtered).

Operator action: review your Deployment/StatefulSet specs for any LD_*, GLIBC_TUNABLES, GCONV_PATH, MALLOC_*, NSS_*, or OPENSSL_* environment variables. If present and intentional, relocate them to the container image. The webhook logs dropped variable names at Info level:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator \
  --tail=200 | grep "envDropped"

H-series summary

Change	PR	Breaking?	Action required
/v1/port/status requires Bearer auth	broker authentication hardening	Yes, if polling without auth	Add Authorization: Bearer <token> header
Unseal token format validation	broker authentication hardening	Possibly, if sending malformed tokens	None for standard deployments
Vault error response sanitization	broker response and environment hardening	Possibly, if parsing vault error details	Parse generic status-code errors; check operator Debug logs for details
JWT scrubbing from responses	broker response and environment hardening	No	None
Dynamic-linker env var stripping	broker response and environment hardening	Yes, if pod specs set LD_* etc. intentionally	Relocate loader env vars to container image

---

Section I: May 2026 -- Admin proxy hardening

Twelve security fixes across four CloudTaser Port PRs (admin proxy and JWT checks, fingerprint and proxy validation, token and comparison hardening, enrollment and role scope checks) harden the port's admin proxy, JWT authentication, enrollment flow, and Helm chart defaults. Six changes are breaking for specific configurations; the remaining six are transparent improvements.

I1: JWT issuer and audience validation

When TRUST_JWT is enabled (--port-trust-jwt=true), the port now validates iss (issuer) and aud (audience) claims in addition to the JWKS signature verification. Tokens missing either claim, or presenting values that do not match the configured expectations, are rejected with a typed ErrJWTInvalidClaims error.

New flags:

Flag	Env var	Purpose
--trust-jwt-issuer	CLOUDTASER_PORT_TRUST_JWT_ISSUER	Expected iss claim value
--trust-jwt-audience	CLOUDTASER_PORT_TRUST_JWT_AUDIENCE	Expected aud claim value

Not-before (nbf) validation is also enforced: tokens presented before their nbf timestamp are rejected with ErrJWTNotYetValid.

When the issuer and audience flags are left empty (the default), claim validation is skipped and only signature verification is performed -- backward compatible with existing deployments.

Operator action: if you use --port-trust-jwt=true with a custom JWKS issuer, ensure your tokens set both iss and aud claims to match the values you pass to --trust-jwt-issuer and --trust-jwt-audience. Tokens that previously passed signature-only validation will now be rejected if the claims do not match.

I2: Trust-mode bypass protection

--port-trust-jwt (env: TRUST_JWT) defaults false. When false, the port validates JWT signatures using its own mTLS client certificate -- no external JWKS endpoint is needed.

Enabling trust-mode without configuring --trust-jwt-jwks now fails closed: the port refuses to start rather than silently accepting unsigned tokens.

Operator action: do not enable --port-trust-jwt=true unless you have a functioning JWKS endpoint and have set --trust-jwt-jwks, --trust-jwt-issuer, and --trust-jwt-audience. Enabling trust-mode without the JWKS URL will prevent the port from starting.

I3: Fingerprint UUID validation

All admin operations that accept a cluster fingerprint (/enroll, /wipe, /init, /token) now validate that the fingerprint is a well-formed UUID. Cluster fingerprints are always UUIDs -- they are derived from the kube-system namespace UID. Non-UUID values are rejected with HTTP 400 before reaching vault.

This closes a path-traversal vector: a crafted fingerprint containing ../ could previously reach vault paths outside the intended cloudtaser/data/clusters/<fingerprint>/ prefix.

Operator action: if your automation sends non-UUID fingerprints (unlikely in standard deployments -- the CLI always sends the real kube-system UID), those requests will now fail with HTTP 400. Ensure all fingerprints are valid UUIDs.

I4: Admin proxy header allow-list

The admin proxy now forwards only an explicit set of allowlisted headers to vault. All other headers -- including custom X-* headers injected by intermediate proxies, load balancers, or service meshes -- are silently dropped before the request reaches vault.

This prevents header-injection attacks where a caller sets vault-meaningful headers (X-Vault-Token, X-Vault-Namespace, X-Vault-Request) to manipulate vault behavior through the admin proxy.

Operator action: if your vault policies depend on custom headers forwarded through the admin proxy, those headers will no longer reach vault. Standard deployments that interact with vault exclusively through the port's API (CLI, operator) are unaffected.

I5: Vault error sanitization

Port error responses to callers no longer include vault internal error details such as vault paths, URLs, or token prefixes. Callers receive a generic error with the HTTP status code:

admin proxy request failed (status 403)

The full vault error is logged at Debug level on the port side.

Operator action: if your automation parses vault error messages returned through the admin proxy, update it to handle the generic format. For debugging, check the port's Debug-level logs directly:

journalctl -u cloudtaser-port --no-pager -n 200 | grep "vaultError"

I6: Constant-time token comparison

The admin wipe endpoint's auth token comparison already used crypto/subtle.ConstantTimeCompare. This PR added comprehensive test coverage confirming the behavior: wrong tokens are rejected, empty admin tokens fail closed, and correct tokens pass through.

When no admin token is configured on the port, all wipe requests are rejected (fail-closed) rather than accepted.

Operator action: none. This is a defense-in-depth improvement with no behavioral change for operators using properly configured admin tokens.

I7: Enrollment token caps

mintEnrollmentToken now creates vault tokens with the following restrictions:

Property	Previous	New
num_uses	Unlimited	3
period	Default	0s (non-periodic)
renewable	Default (yes)	false

The 3-use cap covers: delivery to the wrapper, one retry, and one audit/verification call. This limits the blast radius if an enrollment token leaks during mTLS transit.

Operator action: if you have automation that reuses enrollment tokens for multiple enrollments or renews them, update it to request a fresh token per enrollment. Single-use tokens that were valid before are still valid (1 < 3), but tokens that were reused across many enrollments will exhaust the 3-use cap.

I8: Initialized marker fingerprint scoping and hard-fail

The /initialized marker path is scoped to the cluster fingerprint: cloudtaser/data/clusters/<fingerprint>/initialized. This was already the case in practice, but is now enforced with an AI-CONTRACT marker confirming the design intent.

The behavioral change: marker write failures now hard-fail. When the port cannot write the initialized marker to vault (network error, permission denied), it clears the cert material from the response body before returning it to the caller. This preserves one-time-use replay protection -- previously, a vault 403 on the marker write was silently swallowed, allowing the same init payload to be replayed.

Operator action: ensure vault is reachable and the port's vault token has write access to the marker path during the initialization flow. If the marker write fails, the port returns a response without cert material; the caller must retry the entire init flow.

I9: Dev-mode default false

DEV_MODE (env: CLOUDTASER_PORT_DEV_MODE) defaults false in the ClientConfig zero value and in the CLI flag declaration. Dev mode was never intended for production; this change ensures that misconfigured deployments (accidentally setting DEV_MODE=1 in a production .env file) fail closed rather than bypassing security checks.

validateSecurityFlags enforces mutual exclusion: DevMode=true cannot be combined with EnrollAdminToken (the production auth mechanism).

Operator action: none for production deployments (the default is already correct). If you run the port in dev mode for local testing, the flag continues to work -- you must set it explicitly.

I10: X-Forwarded-For injection protection

extractClientIP now validates all values from X-Forwarded-For and CF-Connecting-IP headers with net.ParseIP. Invalid values -- including path-traversal payloads, UUIDs, and injection attempts -- are rejected, and the function falls back to the raw TCP RemoteAddr.

This prevents per-IP rate limit bypass via header spoofing: an attacker who could set X-Forwarded-For: <unique-value> on each request previously got a fresh rate-limit bucket per forged IP.

Operator action: none. Legitimate proxy chains that set valid IP addresses in X-Forwarded-For are unaffected.

I11: Wildcard service account rejection

boundServiceAccountNames and boundServiceAccountNamespaces in the cloudtaser-openbao Helm chart (values.yaml) now default to explicit values instead of the wildcard "*":

Field	Previous default	New default
boundServiceAccountNames	"*"	cloudtaser-wrapper
boundServiceAccountNamespaces	"*"	default

The Go code already rejected "*" at runtime (since runtime wildcard refusal), so this change closes the chart-level gap -- preventing the wildcard from reaching vault role configuration if the runtime guard were ever bypassed.

Operator action: if you use non-default service account names or namespaces for your wrapper pods, update the boundServiceAccountNames and boundServiceAccountNamespaces values in your values.yaml before upgrading. The wildcard "*" is no longer accepted at either the chart or runtime level.

I12: Anonymous enrollment prevention

The /token endpoint now verifies that the cluster fingerprint is registered in vault (cloudtaser/data/clusters/<fingerprint>/registered) before minting an enrollment token. Unregistered clusters receive HTTP 403.

Previously, calling /token before /enroll would silently succeed -- minting a scoped token for a cluster that vault had no record of. This allowed anonymous enrollment: any caller with network access to the port could obtain a valid vault token without going through the registration flow.

Operator action: ensure your clusters are enrolled (/enroll) before requesting tokens (/token). The standard CLI flow (cloudtaser-cli target install) already does this in the correct order. Automation that called /token directly without prior enrollment will now fail with HTTP 403.

I-series summary

Change	PR	Breaking?	Action required
JWT issuer + audience validation	admin proxy and JWT checks	Yes, if using trust-mode with custom JWKS	Set --trust-jwt-issuer and --trust-jwt-audience
Trust-mode bypass protection	admin proxy and JWT checks	Yes, if enabling trust-mode without JWKS URL	Configure --trust-jwt-jwks before enabling trust-mode
Fingerprint UUID validation	fingerprint and proxy validation	Possibly, if sending non-UUID fingerprints	Ensure fingerprints are valid UUIDs
Admin proxy header allow-list	fingerprint and proxy validation	Yes, if vault policies depend on custom headers	Move vault-meaningful logic off custom proxy headers
Vault error sanitization	fingerprint and proxy validation	Possibly, if parsing vault error details	Check port Debug logs for full errors
Constant-time token comparison	token and comparison hardening	No	None
Enrollment token caps (num_uses=3)	token and comparison hardening	Yes, if reusing enrollment tokens	Request a fresh token per enrollment
Initialized marker hard-fail	token and comparison hardening	Possibly, if vault unreachable during init	Ensure vault reachability during init flow
Dev-mode default false	token and comparison hardening	No (was already the practical default)	None
X-Forwarded-For injection protection	token and comparison hardening	No	None
Wildcard service account rejection	enrollment and role scope checks	Yes, if using non-default SA names	Update boundServiceAccountNames / boundServiceAccountNamespaces
Anonymous enrollment prevention	enrollment and role scope checks	Yes, if calling /token before /enroll	Enroll clusters before requesting tokens

---

Section J: Beacon v0.5.3 — HMAC nonce/timestamp protocol migration

Beacon v0.5.3 changes the HMAC registration protocol. Registrations now include a nonce and RFC3339 timestamp in the HMAC computation, preventing indefinite replay attacks. Old-format registrations (bare HMAC without nonce) are rejected by default in beacon v0.5.3+.

Impact: operator and port clients must be updated to produce new-format registrations before or alongside the beacon upgrade. Mixed environments where the beacon has been upgraded but clients have not will fail to register.

Migration procedure

Step 1. Before upgrading the beacon, set the compatibility window on each beacon instance. Choose a date 2–4 weeks from your upgrade date:

--hmac-compat-until=2026-07-15T00:00:00Z

With this flag set, the beacon accepts both old-format and new-format registrations. Old-format registrations log a warning — use these warnings to verify that clients are migrating correctly before the deadline.

Via Helm:

# values.yaml
hmacCompatUntil: "2026-07-15T00:00:00Z"

Step 2. Upgrade the beacon to v0.5.3+.

Step 3. Upgrade all clients:

• cloudtaser-operator — upgrade to the version that produces nonce+timestamp HMAC registrations (the broker component inside the operator handles beacon registration)
• CloudTaser Port — upgrade to the version that produces nonce+timestamp HMAC registrations

Verify in beacon logs: old-format registration warnings (logged alongside the hmac-compat-until window) should stop appearing once all clients are upgraded.

Step 4. Remove the --hmac-compat-until flag (or let the timestamp expire). Once removed or expired, strict mode is enforced — only new-format registrations are accepted.

Rollback

If issues arise after removing the compatibility window, re-add --hmac-compat-until with a future timestamp. Old-format clients can continue operating until the new deadline, giving time to diagnose and complete the client upgrade.

Summary

Step	Action	Breaking?
Set --hmac-compat-until	Accept both old and new formats with deadline	No
Upgrade beacon	Enforce new format after deadline	Yes for old clients after deadline
Upgrade operator + port	Produce nonce+timestamp HMAC	No for beacon
Remove compat flag	Strict mode; old-format rejected	Yes for any remaining old clients

---

Section K: June 2026 -- Operator JWT audience pinning (v0.13.60+)

Operator v0.13.60 (projected-token authentication) changes how wrapper pods authenticate to the port for secret fetches. The operator now mounts a projected ServiceAccountToken with a dedicated audience and passes it to the wrapper via environment variable. The port must accept tokens with this audience or all secret fetches are rejected.

What changed

The operator mounts a projected ServiceAccountToken at /var/run/secrets/cloudtaser.io/serviceaccount/token with audience cloudtaser.io/secret-fetch. It injects the environment variable CLOUDTASER_SECRET_FETCH_TOKEN_PATH pointing to this path. The wrapper uses this audience-pinned JWT for all secret-fetch authentication through the port, instead of the default Kubernetes service account token.

Why

The default service account token's audience (https://kubernetes.default.svc.cluster.local) is shared with every pod in the cluster. A compromised non-CloudTaser pod could reuse its own default SA token to authenticate to the port and request secrets. Using a dedicated audience (cloudtaser.io/secret-fetch) ensures the port rejects tokens not explicitly minted for the secret-fetch flow -- a compromised pod's default SA token carries the wrong audience and is denied.

Impact

Breaking for port deployments where --trust-jwt-audience is not set to cloudtaser.io/secret-fetch. After upgrading the operator, the wrapper sends JWTs with audience cloudtaser.io/secret-fetch. If the port expects the default Kubernetes audience, it rejects every wrapper JWT with:

invalid claims: audience mismatch: got [cloudtaser.io/secret-fetch], want "https://kubernetes.default.svc.cluster.local"

All wrapper pods will enter CrashLoopBackOff because secret fetches fail.

Operator action for Helm deployments

The Helm chart already defaults port.trustJWT.audience to cloudtaser.io/secret-fetch in values.yaml. No action needed unless you explicitly overrode this value. If you did, update your override:

# values.yaml
port:
  trustJWT:
    audience: "cloudtaser.io/secret-fetch"

Operator action for direct port deployments (non-Helm)

Add the audience flag to port startup:

--trust-jwt-audience=cloudtaser.io/secret-fetch

Or set the environment variable:

CLOUDTASER_PORT_TRUST_JWT_AUDIENCE=cloudtaser.io/secret-fetch

Upgrade order

Port first, then operator

This is the opposite of the usual eBPF → operator → wrapper order. The port must accept the new audience before the operator starts sending audience-pinned JWTs. If the operator upgrades first, all secret fetches fail until the port is configured.

Step 1. Configure the port to accept cloudtaser.io/secret-fetch:

• Helm: verify port.trustJWT.audience is set to cloudtaser.io/secret-fetch (the default). Upgrade the cloudtaser-port chart if needed.
• Direct: add --trust-jwt-audience=cloudtaser.io/secret-fetch to the port process and restart it.

Step 2. Upgrade the operator to v0.13.60+. New and restarted wrapper pods will now use the audience-pinned JWT.

Step 3. Restart protected workloads to pick up the new projected token mount:

kubectl get deployments,statefulsets,daemonsets --all-namespaces \
  -o jsonpath='{range .items[?(@.spec.template.metadata.annotations.cloudtaser\.io/inject=="true")]}{.kind}/{.metadata.namespace}/{.metadata.name}{"\n"}{end}' | \
  while IFS=/ read kind ns name; do
    kubectl rollout restart "${kind,,}/${name}" -n "${ns}"
  done

Verification

After both sides are upgraded, check for audience-mismatch errors:

# Port logs (self-hosted / direct deployment)
journalctl -u cloudtaser-port --no-pager -n 100 | grep -i 'audience'

# Operator logs (Helm deployment)
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator \
  --tail=50 | grep -i 'audience'

No audience mismatch lines should appear. Verify wrapper pods are running and secrets are being fetched:

kubectl get pods -A -l cloudtaser.io/protected=true | \
  grep -v '1/1\|2/2\|3/3\|Running\|Completed' || \
  echo 'all injected pods healthy'

Rollback

If the port is not yet ready to accept the new audience:

• Pin the operator image to a pre-v0.13.60 tag in your Helm values or Deployment spec. Pre-v0.13.60 operators send the default SA token, which the port accepts without audience configuration.
• Or set --allow-legacy-unseal=true on the operator to fall back to the default SA token path.

Once the port is configured, upgrade the operator again.

K-series summary

Change	Component	Breaking?	Action required
Projected SA token with audience cloudtaser.io/secret-fetch	Operator	Yes, if port not configured for audience	Configure port --trust-jwt-audience before upgrading operator
CLOUDTASER_SECRET_FETCH_TOKEN_PATH env var injected	Operator	No (additive)	None
Wrapper uses audience-pinned JWT for secret fetch	Wrapper	No (automatic with new operator)	None
Port must accept cloudtaser.io/secret-fetch audience	Port (cloudtaser-port)	Yes, if audience not configured	Set --trust-jwt-audience=cloudtaser.io/secret-fetch or use Helm default
Helm chart default port.trustJWT.audience	Helm	No (already defaulted)	None unless overridden

---

Section L: September 2026 -- Port-authored per-pod policies, Volume Shield first-use DEKs, Cosign 3 bundle verification

This section covers three changes that ship together: the per-pod policy authoring move, first-use Volume Shield DEK provisioning, and operator verification of Cosign 3 bundle signatures. Upgrade in the order given under Required upgrade order. L4 covers a later CLI change (v0.19.145) to the Port enrollment token role that affects Ports older than v0.4.90.

L1: The port is now the only author of per-pod OpenBao policies

What changed. Per-pod ACL policies (cloudtaser/pods/...) are now written by the port on the EU side. The operator sends the policy request over its per-cluster connection. The port validates it, re-renders it as a read/list-only KV policy, and writes it with its own identity.

The per-cluster operator policy no longer carries any policy-authoring capability. It also loses sudo and the wildcard token-create grants, and it gets read-only access to its own cloudtaser-operator-<fingerprint> policy.

Why. A per-cluster operator token could previously write arbitrary policy content and mint a child token carrying it. A compromised or coerced in-cluster operator could therefore escalate to read every enrolled cluster's secrets. Moving authorship to the port, which runs on your EU infrastructure, closes that path.

Impact: breaking if you skip the order below. The port's own OpenBao policy must grant it the per-pod authoring path before the new port starts:

path "sys/policies/acl/cloudtaser/pods/*" {
  capabilities = ["create", "read", "update", "delete"]
}

If this grant is missing, per-pod policy creation fails closed after the upgrade. The port answers the relayed write with HTTP 502 and the affected pods stay sealed. They are denied rather than admitted with a weaker policy, and there is no fallback to operator-authored policies. cloudtaser-cli source register likewise fails with an explicit Vault 403 message when the port policy lacks the grant. Confirm that the running Port token is bound to the cloudtaser-port policy, then re-run source configure in the mode matching the store's deployment.

Existing enrollments keep their previous (broader) per-cluster operator policy until a port containing this change is running. On startup and on each enrollment it reconciles cloudtaser-operator-<fingerprint> to the narrowed shape, and it refuses relayed operator traffic for a fingerprint until that reconcile succeeds.

L2: Volume Shield first-use DEK provisioning

On first use of an annotated PVC, the Volume Shield sidecar asks the operator broker to provision a data-encryption key.

• Authorization. The operator authorizes the request against the live pod and PVC objects (audience-bound TokenReview, pod UID, ServiceAccount, and a mounted PVC with a matching UID) before contacting the port.
• Generation. The port generates the 32-byte key on the EU side and writes it create-only (KV v2 cas=0). The response is status-only; key material only ever reaches the workload through the normal authorized secret read.
• Port policy. The port's policy narrows to create on cloudtaser/data/volumes/* and read on cloudtaser/metadata/volumes/*. Existing installs keep their broader older grants until reconfigured. Provisioning works under either, because cas=0 never overwrites an existing key.
• Read authorization is unchanged. Provisioning grants no reads: the pod still needs read access to its exact key path, and the port refuses wildcard volume paths in per-pod policies. See Volume Shield installation.

L3: Operator verifies Cosign 3 bundle signatures

CloudTaser release images are signed with Cosign 3, which stores signatures as Sigstore bundles attached through the OCI 1.1 referrers API rather than as legacy .sig tags. Operators before this release could not find those signatures.

With the chart defaults (cosign.enforced=true plus the bundled release key), every wrapper injection was therefore denied. The operator in this release verifies both formats against the configured key. If you disabled cosign.enforced / cosign.requireKey / admission.enforceSignature to work around that denial, re-enable them after upgrading.

L4: CLI v0.19.145 closes the Port enrollment token role

What changed. From CLI v0.19.145, cloudtaser-cli source configure, source install, and source install port write the cloudtaser-operator-enrollment token role in its closed form: allowed_policies=[], with the global cloudtaser-operator policy disallowed. Tokens minted through the role can then carry only per-cluster cloudtaser-operator-<fingerprint> policies, never the global one. Earlier CLI versions write a role that also admits the global policy, and re-running them reopens it.

Impact: breaking for Ports v0.4.53 to v0.4.89. These ports request cloudtaser-operator alongside cloudtaser-operator-<fingerprint> when they enroll an operator. Against a store whose role CLI v0.19.145 or later has written, OpenBao refuses that request (HTTP 400) and the operator stays sealed. Ports v0.4.90 and later request only the per-cluster policy and are unaffected; their bootstrap writes the same closed role.

Action. Upgrade the port to v0.4.90 or later before you run source configure or source install with CLI v0.19.145 or later. The cloudtaser chart 1.0.296 and later digest-pins Port v0.4.110, the version step 2 deploys. Tokens already minted through the old role keep their policies until they expire or are revoked.

Interaction with the order below. Step 1 must still add the L1 grant before a port v0.4.110 or later starts. If your port is older than v0.4.90, run step 1 with CLI v0.19.143 or v0.19.144, which leave the role in the form that port needs, complete step 2, and then re-run the same source configure command with CLI v0.19.145 or later to close the role.

Required upgrade order

1. Update the port's own OpenBao policy to add the grant from L1. First confirm that the running Port token is bound to the named cloudtaser-port policy; source configure reconciles that policy but does not rebind an existing token. Use CLI v0.19.143 or later for a complete existing-install reconciliation, including the named wrapper policy. CLI v0.19.140 can add the Port grant on an in-cluster store, but its target install / target connect writers can restore the wrapper policy's Volume Shield trusted-state grant. For a VM-hosted OpenBao reached by workloads only through the Port, use --kubernetes-auth none; this reconciles the Port policy without creating or changing Kubernetes auth roles:

env -u SECRETSTORE_TOKEN -u VAULT_TOKEN cloudtaser-cli source configure \
  --secretstore-address http://127.0.0.1:8200 \
  --secretstore-token-file /path/to/existing/admin-token \
  --kv-mount cloudtaser \
  --kubernetes-auth none

Use the installation's existing KV mount in place of cloudtaser, and pass its existing custom policy or role names if configured; the example matches the demo-lab VM.

The loopback address applies only when running on the secret-store host. The example clears ambient token variables because they take precedence over --secretstore-token-file. If workload pods authenticate directly to an external OpenBao, use --kubernetes-auth external with --kubernetes-host and --kubernetes-ca-cert-file instead; provide --kubernetes-token-reviewer-jwt-file or grant each workload ServiceAccount system:auth-delegator for self-review. none would leave their Kubernetes auth roles unreconciled. CLI v0.19.140 cannot complete the VM-hosted step because its in-cluster Kubernetes-auth setup requires a pod ServiceAccount CA. For an in-cluster cloudtaser-openbao chart installation, a supported alternative is to re-run the port bootstrap job with cloudtaser-bootstrap v0.4.110 or later. cloudtaser-openbao chart 0.2.4 and later default to bootstrap v0.4.110, pinned by digest. On 0.2.3 and earlier, and on any helm upgrade --reuse-values from those versions (which keeps the old v0.4.102 default), set --set bootstrap.image.tag=v0.4.110 --set bootstrap.image.digest=2b4eacee1841d194c88079aa3b30efe033bcd859a2a2fe7057c10746e7. Chart 0.2.4 refuses to render an older bootstrap image while the bootstrap job is enabled. 2. Then deploy the new port (v0.4.110 or later). 3. Then upgrade the chart / operator and wrappers, and use the new cloudtaser-cli source register for any new clusters. Existing clusters pick up the narrowed per-cluster operator policy when they re-enroll.

Change	Component	Breaking?	Action
Port authors per-pod policies	Port + CLI + operator	Yes, if the port policy lacks the pods grant	Step 1 before step 2
Per-cluster operator policy narrowed (no pods write, no sudo, no wildcards)	Port / CLI	No (applied on re-enrollment)	None
Volume Shield first-use DEK provisioning	Wrapper + operator + port	No (additive; old ports fail closed)	Upgrade the port for first-use provisioning
Cosign 3 bundle verification	Operator	No (fixes denials)	Re-enable signature enforcement if you disabled it
Port enrollment token role closed by CLI v0.19.145 (L4)	CLI	Yes, for Ports v0.4.53–v0.4.89	Upgrade the port to v0.4.90 or later before running source configure / source install with the new CLI

Minimum versions for this section (released 2026-09-27):

Component	Version	Notes
Helm chart cloudtaser	1.0.296	Digest-pins the component images below
Operator	v0.13.172	bundle verification and provisioning relay
Wrapper / wrapper-init	v0.2.181; v0.2.182 for Volume Shield with chart 1.0.299 or later	sidecar DEK requests and broker startup fixes. Chart 1.0.299 and later start the Volume Shield broker in the default fdpass mode without DEK backend settings; brokers older than v0.2.182 exit at startup there. The broker image uses volumeShield.sidecar.image.tag unless volumeShield.broker.image.tag is set: set it to v0.2.182 or later. Chart 1.0.300 pins wrapper v0.2.182
Port	v0.4.110	per-pod policy authoring, DEK provisioning and bootstrap fixes
CLI	v0.19.143 for complete reconciliation	source configure installs the port pods grant, reconciles the named wrapper policy across writers, and supports explicit external/none Kubernetes auth modes
Helm chart cloudtaser-openbao	0.2.4	Bootstrap v0.4.110 by digest; render guard against older bootstrap images

---

Section M: Barrier-backed OpenBao listener PKI

The released OpenBao chart 0.2.17 (umbrella chart 1.0.359) defaults to tls.mode=openbao-pki. Its CA key is generated inside the Secretstore barrier and never exported to Kubernetes. Listener keys are generated in pod memory. Only public certificates enter openbao-tls-trust-bundle and openbao-ca-public ConfigMaps.

Released CLI and separate managed-install gate

The source provision-bao-tls command described here is available from CLI v0.19.149, including the current release v0.19.176. Managed Port installation remains fail-closed until its credential-delivery and authority gates are completed. This OpenBao-only procedure does not enable that path or claim end-to-end CloudTaser acceptance.

The fresh-store lifecycle is:

1. Deploy the signed OpenBao chart with bootstrap disabled. The CLI requires explicit API Service and endpoint CIDRs/ports, keeps NetworkPolicy enabled, and returns after deployment rather than waiting for an uninitialized store to become Ready.
2. Initialize and unseal through verified bootstrap TLS under the administrator's custody. Keep root/unseal material out of Helm values, Kubernetes objects, command arguments and logs. Capture any required recovery shares through the organization's out-of-band custody process.
3. Run cloudtaser-cli source provision-bao-tls with the existing namespace, release name, verified HTTPS leader endpoint, public bootstrap CA file and --secretstore-token-file - (administrator token through stdin). The command verifies an initialized, unsealed store and the default chart PKI bindings, creates an internal CA and finite listener-issuance authority, verifies policy/role/auth readbacks, then publishes and reads back public ConfigMap trust. Missing or mismatched trust, TLS errors and publication failures are errors, never successful provisioning.
4. Independently verify every served listener has changed from its bootstrap certificate to a certificate chaining to openbao-ca-public. Verify renewal changes serial without pod replacement/restart and that health remains available. Test an unlabelled client's denial alongside a labelled client's successful verified-TLS connection. Keep NetworkPolicies in place throughout.

The CLI reads public CA ConfigMaps only; a missing ConfigMap never falls back to a CA Secret. The production CA key remains protected by OpenBao's barrier and administrator-held unseal/recovery authority, not by an etcd Secret.

No legacy upgrade, dual-trust transition or compatibility mode is supported by this procedure. Reprovision a clean store under the approved administrator authority design. The historical Section L describes older releases and is not a current managed-authority enrollment procedure.

Section N: October 2026 -- Direct-vault proxy login token

What changed. In direct-vault mode the operator now projects a ServiceAccount token with audience cloudtaser.io/secret-fetch into the cloudtaser-db-proxy and cloudtaser-s3-proxy sidecars. The token comes from a dedicated volume, cloudtaser-proxy-vault-token, has a one-hour lifetime and is mounted read-only at /var/run/secrets/cloudtaser.io/proxy-vault/token. The operator passes that path in CLOUDTASER_DBPROXY_VAULT_AUTH_SA_TOKEN_FILE or CLOUDTASER_S3PROXY_VAULT_AUTH_SA_TOKEN_FILE, and the proxies present this token at OpenBao Kubernetes auth login instead of the default ServiceAccount token.

Why. The proxies log in with the wrapper's role (cloudtaser.io/secretstore-role, cloudtaser by default). cloudtaser target connect --mode direct binds that role to the cloudtaser.io/secret-fetch audience, so it rejects the default-audience token the proxies used to send, and proxy Transit operations failed.

Impact: version-coupled. The fix takes effect only when both sides are upgraded:

Operator	Proxy image	Login token	Audience-bound role
New	New	Projected cloudtaser.io/secret-fetch	Accepted
New	Old	Default ServiceAccount token (the variable is ignored)	Rejected
Old	New	Default ServiceAccount token (the variable is unset)	Rejected

Neither partial upgrade breaks a setup that works today; it just keeps the old behaviour.

Action required.

1. Upgrade the operator and the proxy images in one change. With Helm, use a chart release whose operator and s3proxy.image / dbproxy.image pins all include this change. If you override those image values, move the overrides to proxy releases that read CLOUDTASER_DBPROXY_VAULT_AUTH_SA_TOKEN_FILE and CLOUDTASER_S3PROXY_VAULT_AUTH_SA_TOKEN_FILE.
2. Recreate pods that carry cloudtaser.io/db-proxy or cloudtaser.io/s3-proxy. Injection happens at admission, so running pods keep the old sidecar spec until they are recreated.
3. Do not weaken the role to work around a partial upgrade. Removing the audience binding lets any default ServiceAccount token in the bound namespaces log in with the wrapper's role.

Newly rejected pods. cloudtaser-proxy-vault-token is a reserved volume name. Admission rejects a direct-vault proxy pod that declares a volume with that name, or that mounts it in any of its own containers, init containers or ephemeral containers.

Section O: October 2026 -- Proxy-only pods are protected pods

What changed. From operator v0.13.190, a pod that carries only an injected cloudtaser-db-proxy or cloudtaser-s3-proxy sidecar, with no wrapper, counts as a CloudTaser-protected pod. The operator's admission gates treat it exactly like a wrapper pod. The pod-security webhook also denies hostPID and hostIPC on every protected pod, including wrapper pods.

A pod is protected when it has any of the following:

• the injected wrapper;
• an operator-injected cloudtaser-db-proxy or cloudtaser-s3-proxy sidecar, in direct-vault or P2P mode;
• secretstore.cloudtaser.io/inject, secretstore.cloudtaser.io/inject, cloudtaser.io/db-proxy or cloudtaser.io/s3-proxy set to "true".

After admission the operator recognises the injected containers themselves. Removing or changing annotations or labels on a running pod does not remove protection.

Why. The proxy sidecars hold OpenBao tokens that can use Transit, the projected cloudtaser.io/secret-fetch login token and, in P2P mode, a cached data key. Before this change, anyone allowed to exec into a proxy-only pod, add an ephemeral debug container or share its process namespace could read them.

Newly rejected requests on proxy-only pods.

Request	Rejected unless
kubectl exec, attach, port-forward, and pods/proxy	the caller's ServiceAccount is listed in cloudtaser.io/allow-exec when the pod is created
kubectl debug (ephemeral containers)	never allowed
shareProcessNamespace: true	cloudtaser.io/allow-share-process-namespace: "true"
hostPID: true or hostIPC: true (all protected pods)	cloudtaser.io/allow-privileged: "true"
Privileged containers, dangerous capabilities	cloudtaser.io/allow-privileged: "true"
Container runtime socket hostPath mounts	never allowed (no opt-out)

The exec, attach, port-forward, proxy and ephemeral-container checks apply to running pods as soon as the operator is upgraded. The pod-spec checks apply when a pod is created. The eBPF readiness gate is unchanged: it still applies only to wrapper pods.

Action required. Before you upgrade, find proxy-only workloads that rely on kubectl exec, kubectl debug or a shared process namespace, and protected workloads (wrapper and proxy-only) that set hostPID or hostIPC. For break-glass access, add the reviewed ServiceAccounts to cloudtaser.io/allow-exec in the pod template, then roll the workload so the annotation is set at creation. Adding or widening the list on a running pod is rejected.

Section P: October 2026 -- eBPF agent socket mount (operator v0.13.190)

What changed. From operator v0.13.190, new wrapper-injected pods mount the eBPF agent socket as an exact-file hostPath of type Socket. If the agent has not created its socket yet, kubelet waits for it. Older operators used an untyped mount, so kubelet created a directory at the socket path when the socket was missing. When the operator runs with --enable-ebpf-gate=false (CLOUDTASER_ENABLE_EBPF_GATE=false; the chart sets this when ebpf.enabled=false or operator.ebpfGate.enabled=false), new pods omit the agent socket volume, its mounts and the agent endpoint settings.

Action required.

1. Keep the eBPF agent deployed before you roll protected workloads, unless you run with ebpf.enabled=false. Agent-enabled deployments still require the socket and enforcement.
2. After upgrading the operator, roll wrapped workloads once (for example kubectl rollout restart) so that they pick up the new volume and environment shape. Existing pod volumes are immutable, so running pods keep the old mount until they are recreated. Recreate Jobs for their next run.
3. If you later turn the eBPF gate on or off, roll wrapped workloads once more for the same reason.

Prerequisites

• helm v3.x installed
• kubectl access to the target cluster
• The cloudtaser Helm chart repository configured:

helm repo add cloudtaser https://charts.cloudtaser.io
helm repo update

---

Upgrade Strategy

The published Helm chart selects a matched set of independently versioned component images:

Component	Chart Value	Current Default
Operator	operator.image.tag	v0.13.209
Wrapper	wrapper.image.tag	v0.2.207
eBPF Agent	ebpf.image.tag	v0.13.164
S3 Proxy	s3proxy.image.tag	v0.2.61

The Helm chart version (currently 1.0.359) tracks independently from component versions. Upgrading the chart may update one or more component images.

---

Zero-Downtime Upgrade

Step 1: Check current versions

helm list -n cloudtaser-system
kubectl get deployment -n cloudtaser-system cloudtaser-operator \
  -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl get daemonset -n cloudtaser-system cloudtaser-ebpf \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

Step 2: Review the new chart version

# Check available chart versions
helm search repo cloudtaser --versions

# Diff the values between current and new chart
helm diff upgrade cloudtaser cloudtaser/cloudtaser \
  -n cloudtaser-system \
  -f values.yaml

Install the helm-diff plugin

helm plugin install https://github.com/databus23/helm-diff -- shows the exact Kubernetes resource diffs before applying.

Step 3: Upgrade the Helm release

transaction_flag=--atomic
if helm upgrade --help | grep -q -- --rollback-on-failure; then
  transaction_flag=--rollback-on-failure
fi

helm upgrade cloudtaser cloudtaser/cloudtaser \
  -n cloudtaser-system \
  -f values.yaml \
  --version <target-chart-version> \
  "$transaction_flag" --wait --timeout 180s

The public installer applies the equivalent transaction automatically (using Helm 4's --rollback-on-failure name when required). It snapshots the existing mutating and validating webhook behavior, keeps both configurations active during the upgrade, waits for the replacement operator and admission endpoint, and restores the prior release and webhook arrays if any gate fails.

Never delete admission webhooks to make an upgrade pass

Deleting either CloudTaser webhook creates an enforcement gap and leaves no safe rollback target. Resolve field ownership in the chart or operator; do not work around it with a pre-upgrade delete. If a transactional upgrade fails, keep the retained Helm log and the still-active prior release for diagnosis.

Step 4: Verify the upgrade

# Check operator is running
kubectl rollout status deployment/cloudtaser-operator -n cloudtaser-system

# Check eBPF daemonset is running on all nodes
kubectl rollout status daemonset/cloudtaser-ebpf -n cloudtaser-system

# Verify webhook is responding
kubectl get mutatingwebhookconfiguration cloudtaser-operator-webhook -o yaml | head -20

---

Component Upgrade Order

When upgrading individual components (setting specific image tags), follow this order:

1. eBPF agent first -- the agent is backward compatible with older wrapper versions. Upgrading it first ensures enforcement is active during the transition.
2. Operator second -- the operator generates injection patches. A new operator version may inject new annotations or environment variables that the wrapper needs.
3. Wrapper last -- wrapper upgrades require pod restarts (the wrapper binary is copied via the init container). The new wrapper image is used on the next pod creation.

Wrapper upgrades require pod restarts

The wrapper binary is copied into each pod's emptyDir volume by an init container at pod creation time. Existing running pods continue using the old wrapper binary until they are restarted. To roll out a new wrapper version:

# Restart all deployments with cloudtaser injection
kubectl get deployments --all-namespaces \
  -o jsonpath='{range .items[?(@.spec.template.metadata.annotations.cloudtaser\.io/inject=="true")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}' | \
  while read dep; do
    kubectl rollout restart deployment "${dep##*/}" -n "${dep%%/*}"
  done

---

CRD Upgrades

The Helm chart templates all four CRDs (CloudTaserConfig, CloudTaserPolicy, SecretMapping, DebugReport) in charts/cloudtaser/templates/crds.yaml behind installCRDs: true (the default). Because the chart uses templated CRDs with a CRD-adopt hook job, helm upgrade does upgrade CRDs -- they are applied as part of the Helm release lifecycle. No separate kubectl apply step is required.

Chart is the canonical CRD source

CRDs are distributed exclusively through the chart at https://charts.cloudtaser.io. Do not apply CRDs from external repositories or unpinned URLs.

---

Certificate Rotation During Upgrade

The operator manages its own webhook and broker mTLS certificates. The certificate lifecycle depends on the operator-level --secret-backend flag (env: CLOUDTASER_OPERATOR_SECRET_STORE), which accepts inmemory (the default), vault, or kubernetes (deprecated alias for inmemory). This is separate from the per-workload spec.secretBackend field in the CloudTaserConfig CRD, which controls secret delivery to wrapped pods.

During an upgrade:

• inmemory (default): Certificates are generated fresh at each pod startup and held in process memory only (secrets_setup.go, inmemory case). No Kubernetes Secrets, no etcd writes, no vault dependency. The 24-hour rotation checker is disabled because certs are ephemeral and regenerated on each restart. Private keys never leave operator process memory.
• vault: Certificates are persisted in the Secretstore (OpenBao/Vault) under cloudtaser/system/{webhook,broker,port}-tls (internal/certs/vault_store.go) and served from process memory. The operator checks certificate expiry every 24 hours and rotates automatically. A legacy code path (internal/certs/rotation.go) writes only public certificate material (ca.crt, tls.crt) to a Kubernetes Secret; tls.key is never written, and any pre-existing tls.key entry is deleted. The private key is written to --cert-dir (default /tmp/k8s-webhook-server/serving-certs, an ephemeral pod filesystem path) only.
• On startup -- regardless of backend -- the operator patches the MutatingWebhookConfiguration and ValidatingWebhookConfiguration directly with the current CA bundle. No Kubernetes Secret is used for webhook CA delivery.

No private key is ever stored in a Kubernetes Secret or etcd. A legacy-path Secret carries public certificate material only.

No manual certificate action is needed during upgrades. The new operator pod generates or loads certificates according to its configured backend and patches the webhook configurations automatically.

---

Rollback

Helm rollback

# List revision history
helm history cloudtaser -n cloudtaser-system

# Roll back to the previous revision
helm rollback cloudtaser -n cloudtaser-system

# Roll back to a specific revision
helm rollback cloudtaser <revision-number> -n cloudtaser-system

Verify rollback

kubectl rollout status deployment/cloudtaser-operator -n cloudtaser-system
kubectl rollout status daemonset/cloudtaser-ebpf -n cloudtaser-system

Rollback considerations

Scenario	Impact	Action
Operator rollback	Existing broker-path Pods can stay sealed when old operator payloads meet newer wrappers. Downgrades can expose the broker key again.	For the current release, see the warning above: recreate on the old wrapper and rotate the CA again; host Port has no supported rotation runbook.
eBPF rollback	Previous enforcement behavior is restored on all nodes	No pod restart needed
Wrapper rollback	Running pods keep the current wrapper. New pods get the old wrapper.	Restart deployments to pick up old wrapper binary
CRD rollback	CRDs are not managed by Helm rollback	Manually reapply old CRD versions if fields were removed

CRDs are not rolled back by Helm

helm rollback does not revert CRD changes. If a CRD was updated with new fields, rolling back the chart leaves the new CRD in place. This is generally safe because CRDs are additive, but verify compatibility.

---

Database Migration Considerations

SaaS Control Plane

The cloudtaser SaaS control plane (cloudtaser-saas) uses an in-memory tenant store. There are no database migrations required when upgrading the SaaS component.

Database Proxy

The database proxy (cloudtaser-db-proxy) does not have its own database. It proxies PostgreSQL connections and performs transparent encryption/decryption. Upgrading the proxy is safe because:

• The encrypted value format is versioned (currently version 1). New proxy versions can always read values encrypted by older versions.
• The encryption key lives in OpenBao Transit, not in the proxy. Key continuity is guaranteed by OpenBao.
• Restart the proxy under planned maintenance and verify readiness, active clients and secret delivery.

However, if a new proxy version changes the encryption format:

1. The new format version byte ensures old values remain readable
2. New writes use the new format
3. Rollback to an older proxy that does not understand the new format will fail to decrypt values written by the new version

Test proxy upgrades in staging first

Deploy the new proxy version in a staging environment and verify both reads (of existing encrypted data) and writes work correctly before upgrading production.

---

Canary Upgrade

For large clusters, consider a canary upgrade using namespace selectors:

Step 1: Deploy the new operator version to a canary namespace

Override the webhook namespace selector to target only the canary namespace:

# canary-values.yaml
operator:
  image:
    tag: "v0.13.209"
wrapper:
  image:
    tag: "v0.2.207"

Step 2: Restart a test workload in the canary namespace

kubectl rollout restart deployment/test-app -n canary

Step 3: Verify protection score

cloudtaser-cli target status -n canary
cloudtaser-cli target audit --secretstore-address https://vault.eu.example.com -n canary

Step 4: Roll out to all namespaces

Once verified, upgrade the Helm release for the full cluster.

---

Emergency: Restore Admission

The webhook template hardcodes Fail. Failure policy is not an effective configurable setting. The schema still accepts compatibility inputs for failure policy and its upgrade guard; schema acceptance does not change the rendered policy. While the operator runs, admission tamper reconciliation restores the expected policy after a live edit. Restore webhook availability instead of relying on a fail-open patch.

If an upgrade causes admission failures, restore the matched component cohort, operator readiness, certificate trust and webhook network access. Verify an admitted test workload before resuming rollout. Use the webhook recovery procedure.

Namespace enrollment bounds the intended scope; removing enrollment or deleting admission configurations does not recover protected secret delivery. Plan and qualify HA and rollback before relying on either during an incident.