Skip to content

kubectl exec into a wrapped pod - what you see, what you don't

cloudtaser changes what ordinary Kubernetes debug commands reveal about a pod's runtime state. This page documents the exact behaviour so SREs, platform engineers, and security reviewers can:

  • Understand what kubectl exec, kubectl debug, kubectl attach, and kubectl logs actually expose against a wrapped pod.
  • Plan debugging workflows that don't require reaching the secret material.
  • Verify the eBPF enforcement policy works as documented against an in-container adversary (not just external compromise).

The admission and runtime gates below concern wrapped/protected containers. The operator's Pod admission policy also covers injected proxy-only Pods; absence of the wrapper annotation alone does not establish that a Pod is unprotected.

Default installations refuse exec before the command starts

The operator denies pods/exec into protected Pods unless cloudtaser.io/allow-exec authorizes the requesting ServiceAccount. Kubernetes RBAC permission alone is insufficient. This admission exception does not bypass the kernel's runtime-bootstrap checks. The same admission gate covers pods/attach, pods/portforward, and pods/proxy. Ephemeral containers have their own admission validation.

Kubelet exec probes use CRI ExecSync directly and do not pass through API CONNECT admission. Current agents support the trusted runtime through kernel-observed executable identity and service lineage, without an extra CAP_SYS_PTRACE grant or /host/proc/1/root enrollment prerequisite. The bootstrap grant ends when the workload command execs. A newly exec'd process still cannot open the protected application's environ or memory under healthy eBPF enforcement.

Check the actual workload probe and runtime-enrollment health. Empty or degraded enrollment must be diagnosed; do not disable enforcement, weaken runtime identity checks, or add capabilities to mask it. Agent readiness alone does not establish compatibility for every workload.

The exec examples below describe only installations where both admission and runtime bootstrap permit the command to start. On default installations expect refusal instead of the command output shown. A refused exec does not test what would happen to a later secret-read syscall.

For validation of the kernel layer, use guest-only node/CRI streaming exec in an isolated test cluster. Bind the current Pod UID and container ID, prove a public command succeeds through CRI streaming, then verify reads of the held application PID's environ and memory are denied without publishing any bytes. Keep default API-exec admission denial as a separate expected result. Do not replace a failed streaming control with ExecSync or treat a generic upgrade error as eBPF proof. This test route does not grant production users node/CRI access or require an admission exception.

Related pages

Two things to separate in your head

Most confusion here comes from conflating two distinct mechanisms:

  1. Application delivery. For eligible glibc-dynamic programs, the wrapper supplies secrets through its interposer and protected file descriptor; they are not in the child's initial environment. Static/musl programs and explicit environment delivery use an environment snapshot instead. Applications can also create ordinary heap copies while using a secret. See Wrapper Design for the delivery paths.

  2. Runtime-created exec processes. When permitted to start, an exec process receives the container spec's environment, not a copy of the wrapped child's live environment or protected delivery channel. The injected spec carries routing configuration, not the fetched secret values.

The separate questions are whether exec can start, how the application receives secrets, and whether a running process can read another process's memory.

The four debug commands, one at a time

kubectl exec <pod> -- env

Shows: the container spec env - CLOUDTASER_* config variables, VAULT_ADDR, PATH, HOSTNAME, standard k8s service discovery variables, plus any non-secret env the PodSpec declared.

Does not show: secret values fetched by the wrapper. They were never in the PodSpec.

Example output fragment:

$ kubectl exec postgres-demo -- env
CLOUDTASER_ORIGINAL_CMD=docker-entrypoint.sh
CLOUDTASER_ORIGINAL_ARGS=postgres
CLOUDTASER_SECRET_PATHS=secret/data/demo/postgres-credentials
CLOUDTASER_ENV_MAP=password=PGPASSWORD,username=PGUSER
VAULT_ADDR=https://vault.eu.example.com:8200
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=postgres-demo
# NO PGPASSWORD here. NO PGUSER value here.

An attacker with exec permissions learns the plumbing (which vault, which paths, which env-var names the app wants) but not the keys.

kubectl exec <pod> -- cat /proc/1/environ

This is the interesting one. The wrapper process (PID 1) does NOT have secrets in its /proc/1/environ snapshot — the kernel froze that snapshot at exec time, when the container was started by the operator webhook with a configuration-only env (CLOUDTASER_*, VAULT_ADDR, etc.). Secrets are fetched from vault into Go heap memory only AFTER the wrapper has already exec'd; os.Setenv calls in the wrapper do not update /proc/PID/environ. The child has secrets in its initial environment only on the environment-delivery path. Interposer delivery avoids that snapshot; applications can still copy values into ordinary memory.

When the command can start and BPF LSM enforcement is active, reads of a monitored process's /proc/<pid>/environ and /proc/<pid>/mem are denied before data is returned. A failed read can report Permission denied; inspect agent events to establish which enforcement path handled it. Do not confuse that with an admission or runc bootstrap refusal that prevented cat from starting.

Kprobe-only enforcement is an explicit reduced-mode opt-in and has raw-path alias limitations. Detection followed by a kill is not proof that no bytes were read. Consult Platform Compatibility and the enforcement table for the deployed mode instead of inferring it from a kernel version alone.

kubectl exec <pod> -- /bin/sh (interactive shell)

Shows: exactly the same env as kubectl exec -- env - container-spec variables only. No secrets.

Caveats for an SRE in the shell:

  • ps -ef works and shows all processes (wrapper, app, shell). No secret material leaks from ps - command-line args from the PodSpec are visible but secrets don't go through args, they go through env.
  • ls /proc/<pid>/ works for listing directory contents.
  • Attempting to cat any of environ, mem, maps, smaps, pagemap of the wrapper or app is blocked as described above.
  • Attempting to attach a debugger (gdb -p <pid>, strace -p <pid>) is blocked via the ptrace kprobe (vector #8).
  • ss -tnp, ss -unp, netstat -tnp work - you can see that the wrapper has an outbound mTLS connection to the beacon, but you cannot inspect the connection contents from inside the pod (TLS).
  • kill -TERM 1 will terminate the wrapper, which propagates to the app and ends the pod. This is not a secret-extraction path.

kubectl attach <pod>

The protected-Pod admission gate applies here too. When attachment is authorized:

Shows: the stdin/stdout/stderr of the main container's PID 1 (the wrapper) - which, after fork+exec, is wired to the application's stdio. Equivalent to kubectl logs -f <pod> plus stdin write access.

Secrets exposure: none, unless the application itself writes a secret to stdout or stderr (an application-level bug caught by eBPF content-matching on write/sendto, vector #9–14).

kubectl logs <pod>

Shows: captured stdout/stderr of the main container.

Secrets exposure: none unless the application logs a secret. If it does:

  • The eBPF layer content-matches on write() and sendto() against known fetched secret values (vector #9). On a match, the write is blocked (kprobe-override) or the process is killed (reactive-kill).
  • Content-matching is not a substitute for not logging secrets - it catches exact matches of known strings, not synonyms or hashes. Treat the eBPF catch as a backstop, not a policy.

Ephemeral containers and kubectl debug

Ephemeral containers and kubectl debug are subject to admission and runtime checks too; do not assume a debug container can start on a protected Pod. If an authorized debug container starts and joins the target process namespace, its environment comes from its own spec, not from the wrapped application.

What this means for the secret:

  • The ephemeral container's processes see their own container spec env - which does not contain any cloudtaser configuration unless explicitly declared in the debug spec. They do not inherit anything from the main container's wrapper or app.
  • Attempts to read /proc/<app_pid>/environ or /proc/<app_pid>/mem from inside the ephemeral container are covered by the target-anchored BPF LSM policy when active. The deployed mode and target enrollment must be verified.
  • With BPF LSM active, unauthorized ptrace access to monitored PIDs is denied. process_vm_readv is blocked only where kprobe override is available; on Tier 2 it is detect-only with reactive kill, which can leave a disclosure window. Check the deployed vector states.
  • A running, authorized ephemeral container can perform only the diagnostics allowed by its namespace placement, permissions and active enforcement profile.

This is the intended security boundary. If you find a path where an ephemeral container can read secrets from a monitored PID on a kprobe-override kernel, file it as a security issue - we want to know.

The debuggability trade-off - named explicitly

You cannot inspect, via any standard Kubernetes mechanism, what secret values a wrapped app currently holds. That's the point. It's also the primary debuggability regression vs. a traditional secrets workflow (Vault Agent writing to a tmpfs file, K8s Secrets mounted as env).

Traditional debug flow: "The app can't connect to the database. Let me kubectl exec and check echo $PGPASSWORD."

cloudtaser debug flow: on default installations the exec is refused. When exec is available, the new process does not inherit the fetched secret. Use these diagnostics instead:

  1. Check the secret was fetched. The wrapper's stdout (captured in kubectl logs) reports each fetch by path, successfully or with an error. If the wrapper couldn't reach OpenBao, this is where you see it. No secret material is logged - only the path and success/failure.

  2. Check the secret contents in OpenBao directly. If you have OpenBao access and the relevant policy, vault kv get secret/data/<path> or bao kv get on the source of truth is the authoritative answer. The wrapper reads what OpenBao returns; if what OpenBao has is wrong, that's where you fix it.

  3. Check the rotation state. Secrets rotate on lease renewal or explicit rotation. Check OpenBao's lease view (vault token lookup, vault list sys/leases/lookup/...) to understand whether the wrapper has an active lease and when it last renewed.

  4. Check wrapper audit logs. The wrapper emits structured logs for every authentication, secret fetch, lease renewal, and rotation event. These land in kubectl logs and, if configured, in your log aggregator.

  5. Use OpenBao audit logging. OpenBao's audit backend logs every secret read with a timestamp, the requester's identity (the pod's ServiceAccount token), and the path. This is the authoritative answer to "did this pod fetch this secret at this time?" - data the eBPF layer doesn't have.

What you cannot do:

  • Read the secret value from within the pod. By design.
  • Attach a debugger to the app and read its memory. Blocked.
  • Treat memfd_secret as a boundary against arbitrary kernel code. It removes the direct-map alias, but does not provide absolute node-root isolation. See Security Model.

This trade-off is the one that makes the sovereignty claim hold. "If the SRE can see it, so can a compromised SRE credential, which is the attack path we're closing." cloudtaser deliberately chooses that side of the trade-off.

The specific debuggability improvements we recommend on your side

If your operations team is used to the "read the env var to diagnose" workflow, plan these around it:

  1. Standardize on OpenBao audit logs as the source of truth for "did this pod receive a secret". OpenBao's audit backend is free, open-source, and produces exactly the structured data your incident-response process needs.

  2. Write operational runbooks that don't require reading secret values. Verify the secret's plumbing (OpenBao has it, the wrapper fetched it, the app received the fetch event in logs) rather than the content. Most bad-credential incidents are fixable from plumbing state alone.

  3. If content inspection is genuinely necessary (e.g., cross-referencing the exact value an app saw against the source of truth), do it at the OpenBao side with a signed audit request - not from inside the cluster. The principle: if a human needs to see a secret value, that access belongs in OpenBao's audit trail, not in a pod's /proc/environ.

  4. For pre-production or staging clusters, consider running pods without the injection annotation so traditional env-var debugging is available. Production clusters keep the annotation on and accept the trade-off.

What a security reviewer should check

For an authorized audit, establish the active enforcement mode before interpreting an error or a killed process as protection evidence:

  • [ ] Check the operator's pods/exec policy and the requesting identity. Without an applicable cloudtaser.io/allow-exec exception, expect admission refusal.
  • [ ] Check kernel-observed runtime enrollment and any degraded state, then verify the wrapped container's actual exec readiness/liveness/startup probe. Do not infer probe success from agent readiness alone.
  • [ ] In a disposable guest, test node/CRI streaming exec separately from API admission. Require a successful public control before environ/mem read-denial checks, with stable Pod/container and application PID/start-time bindings.
  • [ ] Retain exit status and kernel enforcement evidence without publishing secret bytes, lengths or hashes. Check that a new exec process cannot inspect the protected application's environ/mem or use ptrace/process_vm to read it. Do not treat ExecSync, an API denial or a bootstrap failure as this proof.
  • [ ] Apply the same admission, runtime, and target-enforcement checks to an authorized ephemeral-container test; do not assume it is an exec bypass.
  • [ ] Check the agent's actual tier, enforcement vectors and required kernel prerequisites. For BPF LSM protection, bpf must be in the active LSM list; a built-in but inactive BPF LSM is insufficient. Syscall coverage can use the supported override or fmod_ret path; do not infer it from a version alone.

An admission/bootstrap refusal confirms that gate only. A process killed after a read starts does not establish synchronous denial. Report any result inconsistent with the deployed mode and these prerequisites on cloudtaser-ebpf.