Troubleshooting¶

Common issues and their solutions when running cloudtaser. This page covers diagnostics and fixes for the operator, wrapper, eBPF agent, and CLI.

Exec diagnostics on protected containers

Default API pods/exec admission denies kubectl exec into protected Pods. Use Pod status, events, wrapper/agent logs, or an authorized unprotected diagnostic workload. Kubelet exec probes use a separate node/CRI path.

Protected container exec or exec probes fail¶

First distinguish the failing gate:

API kubectl exec is refused: the operator denies CONNECT into protected Pods by design. Ordinary Kubernetes RBAC does not bypass it. A generic unable to upgrade connection can contain an admission refusal; it does not establish a runc or eBPF bootstrap failure. Keep the admission gate enabled.
A kubelet exec probe fails, or runtime namespace setup is denied: check the agent's kernel-observed runtime enrollment, degraded state and actual enforcement vectors. Current agents support kubelet exec probes without an extra CAP_SYS_PTRACE grant or /host/proc/1/root enrollment prerequisite. Empty enrollment or lost bootstrap observations require diagnosis; do not disable enforcement, broaden trusted runtime identity or add capabilities to hide the failure. Verify the workload's own readiness condition after fixing the gap; agent readiness alone is insufficient.

A kernel-layer validation uses node/CRI streaming exec only inside an isolated guest. First require a public control to succeed, then require the protected application's environ/mem reads to fail with explicit permission errors and kernel evidence. API CONNECT denial and ExecSync readiness success are separate results, not substitutes for that test. Never expose node/CRI access to production workloads to make debugging easier.

See kubectl exec Debuggability for admission policy, runtime bootstrap and target-memory enforcement.

Pods Stuck in Init¶

Symptom: Pod stays in Init:0/1 state. The init container that copies the wrapper binary has not completed.

Diagnosis:

kubectl describe pod <pod-name>
kubectl logs <pod-name> -c cloudtaser-init

Common causes:

1. Wrapper image not pullable¶

The init container pulls from europe-west4-docker.pkg.dev/skipopsmain/cloudtaser/cloudtaser-wrapper. Verify that image pull secrets are configured:

kubectl get pod <pod-name> -o jsonpath='{.spec.imagePullSecrets}'

If using a private registry, ensure imagePullSecrets is set in the Helm values.

2. EmptyDir volume mount failure¶

The wrapper is copied to a memory-backed emptyDir at /cloudtaser/. Check that the node has sufficient memory for the volume (the wrapper binary is approximately 10 MB).

3. Operator not running¶

If the operator is down, the mutating webhook may fail and block pod creation (default failurePolicy: Fail):

kubectl get pods -n cloudtaser-system

failurePolicy: Fail blocks pod creation

If the operator is not running, admission fails within the selected namespaces. Restore operator readiness and webhook reachability; follow webhook recovery.

Operator pod Ready does not imply beacon connected (operator v0.9.4+)

The operator's startup probe is decoupled from beacon connectivity. The pod becomes Ready in seconds regardless of whether the port has connected to the beacon yet. If the port has not connected, the operator pod will be Ready but the webhook will return a clear error on pod creation attempts. This is expected during initial bootstrap — wait for the port to connect, then retry pod creation.

Wrapper Cannot Connect to OpenBao¶

Symptom: Pod starts but the application does not receive secrets. Container logs show OpenBao connection errors.

Diagnosis:

kubectl logs <pod-name> -c <container-name> | grep -i vault

Common causes:

1. OpenBao endpoint unreachable¶

Verify network connectivity from the pod to OpenBao:

kubectl exec <pod-name> -- wget -q --spider https://vault.eu.example.com/v1/sys/health

Check firewall rules, VPC peering, security groups, and NetworkPolicies.

2. Kubernetes auth not configured¶

The wrapper authenticates to OpenBao using the pod's ServiceAccount token. Verify the auth method is configured:

cloudtaser-cli target validate \
  --secretstore-address https://vault.eu.example.com \
  --secretstore-token hvs.YOUR_TOKEN

If auth is not configured, run cloudtaser-cli target connect to set it up.

3. Wrong OpenBao role¶

Verify the cloudtaser.io/secretstore-role annotation matches a role configured in OpenBao:

vault read auth/kubernetes/role/cloudtaser

4. ServiceAccount not bound¶

The OpenBao role must allow the pod's ServiceAccount. Check the bound_service_account_names and bound_service_account_namespaces in the OpenBao role.

5. TLS certificate error¶

If OpenBao uses a private CA, the wrapper cannot verify the certificate. Mount the CA bundle into the pod.

Quick connectivity test

Run cloudtaser-cli target validate --secretstore-address https://vault.eu.example.com to check OpenBao health, seal status, and Kubernetes auth configuration in one step.

eBPF Agent Not Starting¶

Symptom: cloudtaser-ebpf pods are in CrashLoopBackOff or not starting.

Diagnosis:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-ebpf
kubectl describe daemonset -n cloudtaser-system cloudtaser-ebpf

Common causes:

1. Kernel too old¶

The eBPF agent requires Linux kernel 5.8+ with BTF (BPF Type Format) support. Check the node kernel:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kernelVersion}{"\n"}{end}'

Kernel version recommendations

For full protection (including memfd_secret), kernel 5.14+ is recommended. See Kernel Compatibility for the complete support matrix.

2. BTF not available¶

The agent needs /sys/kernel/btf/vmlinux. GKE COS and Ubuntu nodes have this by default. Some custom AMIs may not:

kubectl exec -n cloudtaser-system <ebpf-pod> -- ls /sys/kernel/btf/vmlinux

3. Insufficient capabilities¶

The default chart uses UID 0, privileged: false, drops ALL and adds BPF, PERFMON, SYS_RESOURCE, with hostPID and host mounts. Verify node admission permits this exact profile. SYS_ADMIN and privileged mode belong to the explicit legacy profile; do not add them as a general workaround.

4. GKE Autopilot¶

Autopilot clusters do not allow privileged pods or hostPID. Use GKE Standard instead.

5. Fargate (EKS)¶

Fargate does not support DaemonSets or host-level access. Use managed or self-managed node groups.

Unsupported environments

GKE Autopilot and EKS Fargate are unsupported for protected workloads. The required node agent cannot run, and the default admission/binding gate refuses those workloads. No wrapper-only mode is qualified. Use a separately verified node configuration on GKE Standard or EKS node groups; keep enforcement enabled.

Secrets Not Injected¶

Symptom: Application starts but environment variables with secrets are missing.

Diagnosis:

# Check if injection annotation is present
kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations}' | grep cloudtaser

# Check the configured entrypoint for each container
kubectl get pod <pod-name> -o jsonpath='{range .spec.containers[*]}{.name}{": "}{.command}{"\n"}{end}'

Common causes:

1. Missing inject annotation¶

The pod template must have secretstore.cloudtaser.io/inject: "true". The annotation must be on the pod template, not on the Deployment itself:

spec:
  template:
    metadata:
      annotations:
        secretstore.cloudtaser.io/inject: "true"  # HERE, not on the Deployment

2. Wrong env-map syntax¶

The secretstore.cloudtaser.io/env-map annotation maps OpenBao fields to environment variables. Format: vault_field=ENV_VAR,field2=ENV_VAR2:

secretstore.cloudtaser.io/env-map: "db_password=PGPASSWORD,api_key=API_KEY"

Common mistake: reversed order

The correct format is vault_field=ENV_VAR, not ENV_VAR=vault_field. A reversed mapping will silently fail to inject the expected secrets.

3. Wrong OpenBao path¶

Ensure secretstore.cloudtaser.io/secret-paths uses the KV v2 data path:

# Correct:
secretstore.cloudtaser.io/secret-paths: "secret/data/myapp/config"

# Wrong (missing data/ prefix for KV v2):
secretstore.cloudtaser.io/secret-paths: "secret/myapp/config"

4. Namespace not in webhook scope¶

The operator only injects pods in namespaces that are not system namespaces (kube-system, kube-public, kube-node-lease). Verify your namespace is not excluded.

5. Webhook not intercepting¶

Check the MutatingWebhookConfiguration:

kubectl get mutatingwebhookconfiguration cloudtaser-webhook -o yaml

High Latency on Pod Startup¶

Symptom: Pods take significantly longer to start after cloudtaser injection.

Common causes:

1. OpenBao fetch time¶

The wrapper fetches secrets before starting the application. If OpenBao is slow (geographically distant, under load), this adds startup latency.

Reduce OpenBao latency

Deploy OpenBao in the same region as the cluster (still within the EU)
Reduce the number of secret paths per pod (fewer OpenBao API calls)
Ensure OpenBao is not sealed or in standby mode

2. Image entrypoint resolution¶

The operator resolves the container image entrypoint by querying the container registry. This adds latency on the first injection. The result is cached per image.

3. Init container image pull¶

The first pod on a node pulls the wrapper image. Subsequent pods use the cached image. Use imagePullPolicy: IfNotPresent (default) to avoid re-pulling.

Protection Score Low¶

Symptom: cloudtaser-cli target status or cloudtaser-cli target audit reports a low protection score.

The wrapper protection score (max 175 across 17 checks) reports component checks, not complete per-vector enforcement:

Check	Points	Fix
`memfd_secret`	15	Use kernel 5.14+ on nodes
`mlock`	10	Ensure `CAP_IPC_LOCK` is available (or `ulimit -l unlimited`)
`MADV_DONTDUMP`	5	Automatic (requires wrapper v0.0.14+)
`PR_SET_DUMPABLE(0)`	5	Automatic (requires wrapper v0.0.14+)
`guard_pages`	5	Automatic
`buffer_canary`	5	Automatic
Token protected	10	Automatic when memfd_secret or mlock is available
`environ_scrubbed`	5	Automatic (structural property of fork+exec architecture)
`getenv_interposer`	10	Use a glibc-based container image (not Alpine/musl)
eBPF connected	10	Ensure eBPF daemonset is running on the node
`cpu_mitigations`	5	Automatic on modern CPUs
`ebpf_enforce_mode`	15	Deploy eBPF daemonset with `enforceMode: true`
`ebpf_lsm_enforce`	15	Kernel with `CONFIG_BPF_LSM=y` and `bpf` in the boot LSM stack (strongest tier)
Kprobes active	10	Requires kernel `CONFIG_BPF_KPROBE_OVERRIDE` (fallback tier behind LSM)
`confidential_vm`	10	GKE: `--enable-confidential-nodes` with N2D machine type
`vault_tls_verified`	20	Verified TLS certificate chain to OpenBao; configure the trusted CA
`broker_tls_verified`	20	Verified TLS certificate chain to the beacon broker; configure the trusted CA

To improve the score:

Upgrade node kernel to 5.14+ for memfd_secret support (15 points). Review the two TLS checks as well; each contributes 20 points.

Ensure eBPF agent is running on every node (10 points):

kubectl get daemonset -n cloudtaser-system cloudtaser-ebpf

Check kernel kprobe override support (10 points). Note: CONFIG_BPF_KPROBE_OVERRIDE is not enabled on any major cloud provider kernel (GKE, EKS, AKS). See Kernel Compatibility for details.

eBPF Enforcement Issues¶

Symptom: eBPF agent is running but enforcement events are not generated, or legitimate operations are being blocked.

Diagnosis:

# Check agent logs
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-ebpf

# Check enforce mode
kubectl get daemonset -n cloudtaser-system cloudtaser-ebpf \
  -o jsonpath='{.spec.template.spec.containers[0].env}' | grep ENFORCE

Common causes:

1. Enforce mode disabled¶

If ENFORCE_MODE=false, the agent only logs events without blocking. Enable via Helm:

ebpf:
  enforceMode: true

2. Reactive kill fallback¶

On kernels without CONFIG_BPF_KPROBE_OVERRIDE, the vectors that rely on kprobe override have no synchronous deny path unless a BPF LSM hook covers them. With the default chart (ebpf.failClosedOnEnforcementGap true) the agent then returns HTTP 503 from /healthz and /readyz, the install waits and times out, and the operator refuses protected workloads on that node. This is intended behaviour. See Kernel Compatibility for the kernel requirements.

3. Application uses io_uring¶

cloudtaser blocks io_uring_setup() for protected processes because io_uring bypasses buffer-level monitoring. Applications requiring io_uring must use standard syscalls instead.

Ring Buffer Errors¶

Symptom: Agent logs show ring buffer read errors, or the health endpoint reports unhealthy.

Diagnosis:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-ebpf | grep "ring buffer"

Common causes:

1. Consecutive read failures¶

The agent tracks consecutive ring buffer read errors. After 10 consecutive failures, it marks itself unhealthy and stops the event reader:

{"level":"ERROR","msg":"ring buffer reader failed repeatedly, stopping","consecutive_errors":10}

This typically indicates the BPF ring buffer file descriptor has become invalid. Restart the agent pod:

kubectl rollout restart daemonset -n cloudtaser-system cloudtaser-ebpf

2. Event channel full¶

If the agent logs "event channel full, dropping event", the user-space event processing pipeline cannot keep up with kernel event volume. This happens under extreme syscall load from monitored processes.

Mitigation:

Reduce the number of monitored PIDs
Disable LOG_ALL if enabled (it generates events for every syscall, not just violations)
Increase agent CPU limits to allow faster event processing

3. Ring buffer map not found¶

If the agent fails with "events ring buffer map not found", the BPF object file may be corrupted or incompatible with the running kernel. Verify the eBPF object path:

kubectl exec -n cloudtaser-system <ebpf-pod> -- ls -la /opt/cloudtaser/secret_monitor.o

Secret Region Protection Not Working¶

Symptom: The eBPF agent is running but does not detect access to protected process memory.

Diagnosis:

# Check if the agent has registered any protected pod cgroups
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-ebpf | grep "registered"

# Check if the wrapper is connecting to the agent
kubectl logs <pod-name> -c <container-name> | grep "ebpf"

Common causes:

1. Wrapper not registering with agent¶

The wrapper connects to the eBPF agent over the injected Unix socket (unix:///var/run/cloudtaser/agent.sock) to register its PID and secret memory regions. TCP addresses are rejected. If the socket path is not shared between the wrapper and agent, registration fails; inspect wrapper startup errors and readiness.

Verify the host path volume is mounted in both the eBPF daemonset and the protected pod:

kubectl get daemonset -n cloudtaser-system cloudtaser-ebpf \
  -o jsonpath='{.spec.template.spec.volumes}' | python3 -m json.tool

2. BPF maps not populated¶

The agent maintains BPF maps for enforcement. In v0.1.59+ (helm 1.0.61+) the pod-scope map is a BPF_MAP_TYPE_CGROUP_ARRAY; earlier versions used an inode-based monitored_cgroups HASH. The maps relevant for diagnostics:

Map	Purpose
`protected_cgroups`	Cgroup array for pod-scope enforcement (v0.1.59+, replaces `monitored_cgroups` HASH)
`secret_regions`	Memory address ranges containing secrets per PID
`secret_content`	16-byte content prefixes for content-based leak detection

If protected_cgroups is empty (or monitored_pids on pre-v0.1.59 agents), no enforcement occurs. Check agent logs for cgroup registration events.

3. Content-based matching disabled¶

If the agent logs "secret_content map not found, content-based matching disabled" at startup, the BPF object was compiled without the secret_content map. Content-based leak detection (matching secret data in write()/sendto() buffers) will not work, but cgroup-scope region protection still functions.

4. Global privilege escalation detection¶

If GLOBAL_PRIVESC_DETECT=true, the agent monitors kernel module and eBPF program loading from all PIDs on the node, not just protected pod cgroups. If this is disabled and you expect to see module_load or bpf_load events from non-protected processes, enable it:

env:
  - name: GLOBAL_PRIVESC_DETECT
    value: "true"

Webhook Failures¶

Symptom: Pod creation fails with errors referencing the cloudtaser webhook, or pods are created without injection despite having the correct annotations.

Diagnosis:

# Check webhook configuration exists
kubectl get mutatingwebhookconfiguration cloudtaser-operator-webhook

# Check webhook endpoint is reachable
kubectl get endpoints -n cloudtaser-system cloudtaser-operator-webhook

# Check operator logs for webhook errors
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator | grep -i "webhook\|error\|admission"

Common causes:

1. Certificate expired or CA bundle mismatch¶

For the single-operator startup path, certificate storage is inferred from the backend implementation:

Backend	Webhook certificate home (inferred from startup code)
`inmemory`	Process memory; a fresh CA and leaf are generated on startup.
`vault`	Vault/OpenBao certificate store, loaded into process memory; a store initialization failure serves a temporary denial listener while initialization retries.

The deprecated backend name kubernetes is normalized to inmemory. The public CA bundle in the admission configurations must match the serving certificate. Diagnose certificate errors without reading private key material:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator | grep -i "cert\|tls\|webhook"

For the in-memory backend, restart the operator to regenerate ephemeral webhook material, restore Port introduction and verify admission and delivery. For the direct Vault backend, restore store connectivity and inspect the persisted certificate lifecycle; restarting does not force CA replacement. See certificate recovery.

2. Namespace not in webhook scope¶

The chart opts namespaces in with a label and excludes system namespaces:

Selector component	Inferred chart value
Namespace opt-in	`cloudtaser.io/inject=true`
Namespace name exclusions	`[<install namespace>, "kube-system"]`
Additional system namespace exclusions	`request.namespace != 'kube-system' && !request.namespace.startsWith('gke-')`

An injection annotation alone does not enroll a namespace. The install namespace, kube-system and names beginning with gke- remain outside this webhook scope.

Fix: Use an eligible workload namespace and enroll it explicitly:

kubectl label namespace <workload-namespace> cloudtaser.io/inject=true --overwrite

Verify both the namespace label and the installed webhook selector before checking the pod's injection annotations.

3. Webhook timeout¶

The webhook has a default timeout of 10 seconds (operator.webhook.timeoutSeconds in Helm values). If the operator is under heavy load or the container registry is slow (the operator resolves image entrypoints by querying the registry), the webhook may time out.

Diagnosis:

# Check for timeout events
kubectl get events --field-selector reason=FailedCreate -A | grep cloudtaser

Fix: Increase the webhook timeout in Helm values:

operator:
  webhook:
    timeoutSeconds: 30

Or investigate why the operator is slow (check CPU/memory limits, registry latency).

4. Annotation format errors¶

The operator silently skips injection if annotations are malformed. Common mistakes:

Mistake	Correct
`secretstore.cloudtaser.io/inject: true` (no quotes)	`secretstore.cloudtaser.io/inject: "true"`
Annotations on Deployment metadata	Annotations on `spec.template.metadata`
`secretstore.cloudtaser.io/env-map: "ENV_VAR=vault_field"`	`secretstore.cloudtaser.io/env-map: "vault_field=ENV_VAR"`
Missing OpenBao address	Set `cloudtaser.io/secretstore-address` or use a `CloudTaserConfig` CR
Missing secret paths	Set `secretstore.cloudtaser.io/secret-paths`

Diagnosis:

# Check if the pod has the injection status annotation
kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations.cloudtaser\.io/status}'

If the annotation is missing, the webhook did not process the pod. Check if the inject annotation is on the pod template (not the Deployment itself).

5. Webhook outage blocks pods in selected namespaces¶

The webhook template hardcodes Fail. Failure policy is not an effective configurable setting. The schema still accepts compatibility inputs for failure policy and its upgrade guard; schema acceptance does not change the rendered policy. While the operator runs, admission tamper reconciliation restores the expected policy after a live edit. Restore webhook availability instead of relying on a fail-open patch.

The outage refuses Pod CREATE/UPDATE requests within the namespace selector scope; see operational readiness.

Recovery: Restore operator readiness, Service endpoints, certificate trust and control-plane network access, then retry admission. Plan and qualify HA before relying on failover. Namespace enrollment bounds the intended injection scope; removing enrollment also removes that protection and is not a recovery step for workloads that need secrets. Do not use an unqualified policy edit to resume protected workloads.

6. Port not yet connected to beacon¶

Symptom: The operator pod is Running and Ready, but pod creation fails with a webhook error indicating the broker is not connected or the port is unavailable.

This is expected during initial installation: the operator's startup probe is decoupled from beacon connectivity (operator v0.9.4+). The pod becomes Ready quickly, but the beacon P2P match between the operator and the port completes asynchronously — typically within 500ms once both sides have connected, but only after the port has registered with the beacon.

Diagnosis:

# Check operator logs for beacon connection status
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator | grep -i "beacon\|port\|broker"

Look for log lines indicating the operational mTLS connection is established. If you see only init-phase messages and no operational connection, the port has not yet connected.

Common causes:

The port (cloudtaser-port) is not running or has not reached the beacon yet.
The beacon address in the Helm values (operator.broker.beacon.address) does not resolve to the correct host.
Network connectivity from the port side to the beacon is blocked.

Fix: Verify the port is running and can reach the beacon, then retry pod creation. Once the port connects, the P2P match completes in under 500ms and subsequent webhook calls succeed.

7. Multiple operator replicas¶

Do not diagnose admission TLS by looking for a certificate Secret. HA uses a separate admission trust lifecycle. Check each replica's readiness and logs, the public admission CA bundle and the selected backend before changing replica count. A restart does not establish that failover or trust propagation has been qualified.

Operator Cannot Connect to OpenBao¶

Symptom: The operator pod starts but crashes or logs errors about OpenBao connectivity. The webhook does not serve TLS and pod creation may be blocked.

Diagnosis:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator | grep -i "vault\|secret backend"

Common causes:

1. OpenBao address not configured¶

The chart default is inmemory (inferred from values). This section applies to explicit legacy direct secretBackend: vault, which requires operator.vault.address (or its operator.secretstore.address alias). If the address is empty, the operator cannot connect.

Fix: Set the OpenBao address in Helm values:

operator:
  vault:
    address: "https://vault.eu.example.com"

Or set the environment variable VAULT_ADDR on the operator deployment.

2. Kubernetes auth not configured in OpenBao¶

The operator authenticates to OpenBao using Kubernetes auth. If the auth method is not enabled or the cloudtaser-operator role does not exist, authentication fails.

Diagnosis:

# Check if Kubernetes auth is enabled
vault auth list | grep kubernetes

# Check if the operator role exists
vault read auth/kubernetes/role/cloudtaser-operator

Fix: Run cloudtaser-cli source configure to create the required policy and auth role:

cloudtaser-cli source configure \
  --openbao-addr https://vault.eu.example.com \
  --token hvs.YOUR_ADMIN_TOKEN

3. ServiceAccount not bound to OpenBao role¶

The OpenBao Kubernetes auth role must allow the operator's ServiceAccount. If the ServiceAccount name or namespace does not match the role's bound_service_account_names and bound_service_account_namespaces, authentication is rejected.

Diagnosis:

vault read auth/kubernetes/role/cloudtaser-operator

Check that bound_service_account_names includes the operator's ServiceAccount name and bound_service_account_namespaces includes cloudtaser-system.

4. OpenBao is sealed¶

If OpenBao is sealed, all requests fail. The operator will log connection errors and retry.

Fix: Unseal OpenBao:

vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

5. Network connectivity¶

The operator pod must be able to reach OpenBao over the network. Common blockers include NetworkPolicies, firewall rules, VPC peering, and security groups.

Diagnosis:

kubectl exec -n cloudtaser-system deploy/cloudtaser-operator -- \
  wget -q --spider https://vault.eu.example.com/v1/sys/health

6. OpenBao TLS certificate not trusted¶

If OpenBao uses a private CA, the operator cannot verify the TLS certificate. Mount the CA certificate or set vault.caCert in Helm values. For development only, vault.tlsSkipVerify: true disables verification.

Do not use tlsSkipVerify in production

Disabling TLS verification allows man-in-the-middle attacks on the connection between the operator and OpenBao. Use it only for local development.

7. Falling back to in-memory secrets¶

If OpenBao connectivity cannot be restored immediately and pod creation is blocked, switch the operator to in-memory secrets as a temporary fallback:

helm upgrade cloudtaser cloudtaser/cloudtaser \
  --namespace cloudtaser-system \
  --set operator.secretBackend=inmemory \
  --reuse-values

The operator will restart, generate certificates, hold them in memory, and resume serving the webhook. Switch back to vault after resolving OpenBao connectivity. See Zero Kubernetes Secrets Architecture for details on OpenBao-based secret backend.

Wrapper Startup Failures¶

Symptom: Pod starts but the application container exits immediately or logs errors from the cloudtaser wrapper.

Diagnosis:

# Check wrapper startup logs
kubectl logs <pod-name> -c <container-name> 2>&1 | head -50

# Check if the wrapper binary exists
kubectl exec <pod-name> -c <container-name> -- ls -la /cloudtaser/wrapper

Common causes:

1. Missing required environment variables¶

The wrapper validates its configuration at startup and exits with an error if required variables are missing. Required variables:

Variable	Source
`VAULT_ADDR`	From `cloudtaser.io/secretstore-address` annotation or `CloudTaserConfig` CR
`CLOUDTASER_SECRET_PATHS`	From `secretstore.cloudtaser.io/secret-paths` annotation
`CLOUDTASER_ORIGINAL_CMD`	Resolved by operator from container image or pod spec
`CLOUDTASER_ENV_MAP`	From `secretstore.cloudtaser.io/env-map` annotation

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "configuration error"

Typical error messages:

configuration error: VAULT_ADDR is required
configuration error: CLOUDTASER_SECRET_PATHS is required (comma-separated list)
configuration error: CLOUDTASER_ORIGINAL_CMD is required
configuration error: CLOUDTASER_ENV_MAP is required

Fix: Ensure all required annotations are set on the pod template:

annotations:
  secretstore.cloudtaser.io/inject: "true"
  cloudtaser.io/secretstore-address: "https://vault.eu.example.com:8200"
  cloudtaser.io/secretstore-role: "myapp"
  secretstore.cloudtaser.io/secret-paths: "secret/data/myapp/config"
  secretstore.cloudtaser.io/env-map: "password=PGPASSWORD,api_key=API_KEY"

2. OpenBao authentication failure¶

The wrapper authenticates to OpenBao using the pod's Kubernetes ServiceAccount token. This can fail for several reasons.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep -i "vault\|auth\|login"

Common errors and fixes:

Error	Cause	Fix
`kubernetes login: ... connection refused`	OpenBao endpoint unreachable	Check firewall rules, VPC peering, NetworkPolicies
`kubernetes login: ... 403 Forbidden`	Role not configured or SA not bound	Verify OpenBao role exists and allows the pod's ServiceAccount
`kubernetes login returned no auth info`	Auth method disabled or wrong mount path	Check `vault auth list` and the `cloudtaser.io/secretstore-auth-path` annotation
`reading service account token: no such file`	SA token not mounted	Check `automountServiceAccountToken` is not `false`
`TLS handshake error`	OpenBao uses a private CA	Mount the CA bundle or set `cloudtaser.io/secretstore-tls-skip-verify: "true"` (dev only)

3. Secret fetch failure¶

After authentication, the wrapper fetches secrets from the configured paths.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "failed to fetch secrets"

Common errors:

Error	Cause	Fix
`secret not found: secret/data/myapp/config`	Wrong path or secret does not exist	Verify path with `vault kv get secret/myapp/config`
`permission denied`	OpenBao policy does not allow reading this path	Update the OpenBao policy for the cloudtaser role
`reading secret: ... 404`	Missing `data/` prefix for KV v2	Use `secret/data/myapp/config` not `secret/myapp/config`

4. Env map parse failure¶

The CLOUDTASER_ENV_MAP format is vault_field=ENV_VAR. The wrapper exits if it cannot match OpenBao fields to the fetched secrets.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "failed to parse env map"

Fix: Verify that the field names in the env map match the keys in OpenBao secret:

# Check what keys exist in the secret
vault kv get -format=json secret/myapp/config | jq '.data.data | keys'

# Env map must reference these exact field names
# secretstore.cloudtaser.io/env-map: "password=PGPASSWORD,username=PGUSER"

5. Original command resolution failure¶

The operator resolves the container's original entrypoint (command + args) and sets it as CLOUDTASER_ORIGINAL_CMD and CLOUDTASER_ORIGINAL_ARGS. If the container image specifies neither ENTRYPOINT nor CMD, the operator queries the container registry.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "CLOUDTASER_ORIGINAL_CMD"

If CLOUDTASER_ORIGINAL_CMD is empty:

The operator could not resolve the entrypoint from the pod spec or the container registry
Check that imagePullSecrets are configured (the operator needs registry access)
Set the command explicitly in the pod spec

6. memfd_secret or mlock failure¶

The wrapper allocates protected memory for secrets. If CLOUDTASER_REQUIRE_MEMFD_SECRET=true or CLOUDTASER_REQUIRE_MLOCK=true is set but the kernel does not support the feature, the wrapper exits.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep -i "memfd_secret\|mlock\|FATAL"

Fix: Either upgrade the node kernel (5.14+ for memfd_secret) or remove the requirement flags. Without these flags, the wrapper logs a warning but continues with degraded protection.

7. Sealed mode: waiting for unseal¶

If cloudtaser.io/secretstore-auth-method: "token" is set but no token is provided, the wrapper starts in sealed mode. It is alive (liveness probe passes) but not ready (readiness probe returns 503) until a token is delivered via POST /v1/unseal from the operator's auth broker.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "sealed"

Look for:

{"level":"INFO","msg":"sealed: token auth selected but no token provided, waiting for POST /v1/unseal"}

Fix: Ensure the operator's auth broker is running and can reach the pod. The broker delivers tokens to sealed pods automatically. Check operator logs:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator | grep "broker\|unseal"

8. Lease renewal failure¶

After initial secret fetch, the wrapper renews OpenBao leases periodically (default: every 30 seconds). If renewal fails, the wrapper re-fetches the secret.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "lease renewal\|token renewal"

If both renewal and re-fetch fail repeatedly, the application continues running with the last known secrets. The wrapper logs errors but does not kill the child process.

9. Rotation strategy errors¶

The wrapper supports three rotation strategies, applied when it re-fetches secrets (a re-fetch after a failed leased-secret renewal, or SIGHUP to the wrapper; static KV-v2 changes are not detected automatically, see annotations):

Strategy	Behavior
`restart`	Kills and re-forks the child process with new env vars (default; works for every binary)
`sighup`	Sends SIGHUP to the child; relies on the application to re-read its secrets on the signal
`none`	Logs the change but does nothing

Choice of strategy is per-workload. restart is the default and is compatible with every binary (static Go, musl, any libc) because it simply re-executes the child with the new environment. sighup is only appropriate for applications that have an explicit SIGHUP handler for config reload (nginx, postgres, some custom daemons) -- for most off-the-shelf workloads, leave the default.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "rotation\|SIGHUP"

Common CLI Errors¶

"failed to build kubeconfig"¶

The CLI cannot find or parse your kubeconfig. Ensure ~/.kube/config exists or pass --kubeconfig:

cloudtaser-cli target status --kubeconfig /path/to/kubeconfig

"OpenBao is sealed"¶

The OpenBao instance is sealed and cannot serve requests. Unseal it:

vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

"permission denied" on OpenBao operations¶

OpenBao token used with cloudtaser-cli target connect requires admin-level permissions. Ensure it has policies for:

sys/auth/* (to enable and configure auth methods)
auth/kubernetes/* (to configure Kubernetes auth)
sys/policies/* (to create policies)

CrashLoopBackOff with "context deadline exceeded"¶

Symptom: Wrapper pod enters CrashLoopBackOff. Logs show context deadline exceeded during the broker secret fetch or OpenBao authentication.

Diagnosis:

kubectl logs <pod-name> -c <container-name> 2>&1 | grep "context deadline exceeded"

Common causes:

1. NetworkPolicy blocking DNS on GKE (NodeLocal DNSCache)¶

On GKE clusters with NodeLocal DNSCache enabled, DNS traffic goes to 169.254.20.10 (a hostNetwork address), not to the kube-dns ClusterIP. A NetworkPolicy egress rule using namespaceSelector: kube-system does not match hostNetwork pods — DNS is silently blocked and all name resolution times out.

Fix: Add an explicit ipBlock egress rule for 169.254.20.10/32 on port 53 in your NetworkPolicy. See the full explanation and YAML in GKE Deployment Guide — NetworkPolicy and NodeLocal DNSCache.

2. OpenBao unreachable¶

The broker or OpenBao endpoint is down or unreachable from the pod network. Check connectivity:

kubectl exec <pod-name> -- wget -q --spider --timeout=5 https://vault.eu.example.com/v1/sys/health

3. Beacon not matched¶

If using P2P mode, the beacon may not have matched the operator and port yet. Check operator logs for beacon connection status. See Port not yet connected to beacon above.

Getting Help¶

Check component logs:

kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-operator
kubectl logs -n cloudtaser-system -l app.kubernetes.io/name=cloudtaser-ebpf
kubectl logs <pod-name> -c <container-name>

Run validation:

cloudtaser-cli target validate --secretstore-address https://vault.eu.example.com

Run a full audit:

cloudtaser-cli target audit --secretstore-address https://vault.eu.example.com