Skip to content

Wrapper Design

Authorised application boundary

Authorised application code is trusted. An authorised app can send out a secret it has copied or encoded. Current content checks do not provide a general guarantee against this disclosure or a malicious authorised app. Memory-read tests and a Ready agent do not prove copied/encoded-secret exfiltration prevention. Enforced NetworkPolicy can restrict destinations, but allowed traffic remains a channel. See Security Model.

The wrapper runs as PID 1 of the application container. It authenticates to the configured secret store, fetches secrets, and fork+execs the original application command. How the application receives those secrets depends on its binary and environment API. Eligible glibc-dynamic programs use the LD_PRELOAD interposer by default: secret keys are removed from the exec environment and served through getenv() from a protected memory mapping. Static binaries and musl environments use inherited environment delivery instead.

Leadership summary

The wrapper delivers secrets into application memory without putting their values in the Pod specification or Kubernetes Secrets. Runtime compatibility requires choosing the delivery mode that matches the application's environment API. In particular, Python's os.environ needs an explicit snapshot mode; restart rotation refreshes startup snapshots.

Environment delivery has different exposure and inheritance properties

Static/musl fallback and CLOUDTASER_KEEP_SECRETS_IN_ENV=true put secret values in the child's exec environment. Descendants can inherit those values, and the kernel-managed /proc/<pid>/environ contains them. Active eBPF enforcement restricts external reads; it does not remove the values or prevent an authorized application from logging or passing them on.

Strict interposer delivery avoids placing mapped secret values in the exec environment, but an application that snapshots environ does not obtain them through its native snapshot API. Use the runtime compatibility table to select a mode. Merely compiling Go with CGO does not change os.Getenv into a libc getenv() call.

Why Fork+Exec

Applications expect secrets in environment variables. Postgres reads PGPASSWORD. Redis reads REDIS_PASSWORD. Node apps read DATABASE_URL. This is the universal convention.

The alternatives each have problems:

  • JSON files on tmpfs force every customer to modify their application to read from a custom path in a custom format. This breaks the "application code compatibility subject to the runtime delivery contract" promise and makes it impossible to wrap off-the-shelf software like postgres, mysql, or nginx without custom entrypoint scripts.
  • Real environment variables in the container spec expose secrets through the Kubernetes API, etcd, container runtime inspection, and cloud provider audit logs. This violates the data sovereignty guarantee.

Fork+exec launches the application in every mode. The secret delivery modes differ:

  • Strict interposer delivery (default for compatible non-shell applications when the interposer is active): the wrapper removes mapped secret keys from the child's exec environment. The interposer serves getenv()/secure_getenv() reads from the shared memory mapping. This mode does not populate runtime environment snapshots.
  • Snapshot strip mode: CLOUDTASER_ENV_SNAPSHOT_MODE=strip populates the process environment at library initialization, before the interpreter takes its snapshot. Builds with shell auto-selection use this mode automatically for recognized shell entrypoints; other snapshot runtimes require explicit selection. Interposed exec-family calls strip mapped keys from descendant envp. Snapshot values remain plaintext in application memory and do not refresh on SIGHUP.
  • Inherited environment delivery: static binaries and musl environments use this fallback; CLOUDTASER_KEEP_SECRETS_IN_ENV=true also retains actual environment entries for dynamic programs. Descendants inherit those entries unless the application removes them. Use restart rotation to refresh startup values.

Keep eBPF enforcement enabled in every mode. See Secret Delivery Mode for the settings, exec-hook coverage, and inheritance trade-offs.

How It Works

Using a vanilla postgres container as an example:

1. Pod starts
2. Init container copies the wrapper binary to a shared volume
3. Postgres container starts with modified entrypoint:
   command: ["/cloudtaser/wrapper"]
   env:
     - CLOUDTASER_ORIGINAL_CMD=docker-entrypoint.sh
     - CLOUDTASER_ORIGINAL_ARGS=postgres
     - VAULT_ADDR=https://vault.eu.skipops.io:8200
     - CLOUDTASER_SECRET_PATHS=secret/data/prod/db
     - CLOUDTASER_ENV_MAP=password=PGPASSWORD,username=PGUSER

4. Wrapper (PID 1 of the container):
   a. Authenticates to the EU secret store via Kubernetes service account
   b. Fetches secrets from configured paths
   c. Prepares the interposer mapping or the fallback child environment
   d. Removes wrapper configuration variables from the child environment
   e. fork+execs the original command (docker-entrypoint.sh postgres)

5. Child process (docker-entrypoint.sh -> postgres):
   a. Starts with the selected delivery mode
   b. Postgres calls getenv("PGPASSWORD")
   c. Gets the value through the interposer or inherited environment
   d. Postgres authenticates. Works normally.

6. Wrapper stays alive as parent:
   a. Renews vault token periodically
   b. Renews secret leases
   c. On secret rotation: kills child, re-fork+execs with new env
   d. Forwards signals to child (SIGTERM, SIGINT, etc.)

Postgres can use its existing getenv() access. If an entrypoint shell itself reads a mapped secret as $VAR, that shell needs a snapshot-compatible delivery mode too; check the complete entrypoint chain, not only the final server binary.

Component Architecture

+---------------------------------------------------------+
|  Application Pod                                         |
|                                                          |
|  +----------------------------------------------------+  |
|  |  Application Container                              |  |
|  |                                                     |  |
|  |  /cloudtaser/wrapper (PID 1)                       |  |
|  |    |                                                |  |
|  |    +-- Authenticates to EU vault                    |  |
|  |    +-- Fetches secrets                              |  |
|  |    +-- Prepares the selected delivery mode          |  |
|  |    +-- Strips CLOUDTASER_* vars from env           |  |
|  |    +-- Manages token + lease renewal                |  |
|  |    +-- Handles secret rotation (re-fork+exec)       |  |
|  |    |                                                |  |
|  |    +-- fork+exec --> original command (PID 2+)     |  |
|  |                        |                            |  |
|  |                        +-- Uses the selected mode   |  |
|  |                        +-- Runs unmodified          |  |
|  +----------------------------------------------------+  |
|                                                          |
|  +------------------------+  +------------------------+  |
|  |  Shared Volume         |  |  eBPF DaemonSet        |  |
|  |  (emptyDir)            |  |  (node-level)          |  |
|  |                        |  |                        |  |
|  |  /cloudtaser/         |  |  Prevents:             |  |
|  |    wrapper (binary)    |  |  - ptrace attach       |  |
|  |                        |  |  - /proc/pid/mem reads |  |
|  |                        |  |  - /proc/pid/environ   |  |
|  |                        |  |    reads by others     |  |
|  |                        |  |  - core dump writes    |  |
|  +------------------------+  |  - secret exfiltration |  |
|                              +------------------------+  |
+---------------------------------------------------------+
          |
          | mTLS / Kubernetes auth
          v
+----------------------+
|  EU OpenBao / Vault  |
|  (EU-hosted only)    |
+----------------------+

Wrapper Binary (Go)

The wrapper is a Go binary that runs as PID 1 of the application container. It is the container's entrypoint, rather than a sidecar. Dynamic secret delivery also uses the native interposer. Responsibilities:

  • OpenBao authentication: Kubernetes service account auth against EU-hosted OpenBao. The operator may provide a short-lived token instead.
  • Secret fetching: Reads secrets from configured OpenBao paths.
  • Environment preparation: Selects interposer or inherited environment delivery and strips wrapper configuration variables from the child environment.
  • Process execution: fork+exec of the original container command with the selected delivery configuration.
  • Signal forwarding: Forwards SIGTERM, SIGINT and other signals to the child process. SIGHUP is handled by the wrapper itself: it re-fetches every configured secret path and applies the rotation strategy (only the sighup strategy relays a SIGHUP to the child).
  • Lease management: Renews OpenBao token and secret leases on a configurable interval.
  • Secret rotation: When the wrapper re-fetches secrets (SIGHUP, or a failed lease renewal), applies the rotation strategy, by default killing the child process and re-fork+exec'ing it with an updated environment. Direct-Vault mode also checks static KV-v2 versions; broker/beacon mode requires an explicit fetch trigger, as described below.
  • Health endpoint: HTTP server on configurable port for Kubernetes liveness/readiness probes.

eBPF Enforcement Layer

The eBPF daemonset runs at the node level and enforces runtime protections for all cloudtaser-injected pods:

  • Block ptrace attach: Prevents debuggers, gcore, strace from reading process memory containing secrets
  • Block /proc/pid/mem reads: Prevents direct memory reads by other processes
  • Block /proc/pid/environ reads: Prevents other processes from reading the child's environment, which contains the secrets. This is the critical enforcement -- without it, any process in the same pid namespace could read /proc/<child_pid>/environ
  • Block core dumps: Prevents core dump generation that could include secret environment data
  • Detect heap dump tools: Intercepts Java jmap, Go profiling tools, and other memory dump utilities

Entrypoint Resolution

The operator webhook must know the container's original command and args to pass to the wrapper as CLOUDTASER_ORIGINAL_CMD and CLOUDTASER_ORIGINAL_ARGS.

Case What the webhook does
command and args are explicit in pod spec Read them directly from the submitted spec
command or args are missing (image defaults) Query the container registry API to read ENTRYPOINT and CMD from the image config

The operator uses image pull secrets from the pod spec (or service account) to authenticate to private registries. It caches image metadata to avoid repeated registry calls for the same image digest.

This auto-detection is critical for the "zero changes" promise: customers using stock images with default entrypoints (postgres, nginx, redis) do not need to specify command in their pod specs.

Pod Mutation Details

The webhook applies the following mutations to matching pods:

Add init container:

initContainers:
  - name: cloudtaser-init
    image: europe-west4-docker.pkg.dev/skipopsmain/cloudtaser/cloudtaser-wrapper:<version>
    command: ["sh", "-c", "cp /cloudtaser-wrapper /cloudtaser/wrapper"]
    volumeMounts:
      - name: cloudtaser-bin
        mountPath: /cloudtaser
    resources:
      limits:
        cpu: 50m
        memory: 32Mi

Modify application container:

containers:
  - name: <original-name>
    command: ["/cloudtaser/wrapper"]
    env:
      - name: CLOUDTASER_ORIGINAL_CMD
        value: <resolved command>
      - name: CLOUDTASER_ORIGINAL_ARGS
        value: <resolved args>
      - name: VAULT_ADDR
        value: <from annotation or operator config>
      - name: CLOUDTASER_SECRET_PATHS
        value: <from annotation>
      - name: CLOUDTASER_ENV_MAP
        value: <from annotation>
    volumeMounts:
      - name: cloudtaser-bin
        mountPath: /cloudtaser

Add shared volume:

volumes:
  - name: cloudtaser-bin
    emptyDir:
      medium: Memory
      sizeLimit: 8Mi

The init container stages the wrapper and libcloudtaser.so in the memory-backed emptyDir at /cloudtaser/. The wrapper selects interposer delivery for eligible glibc-dynamic programs. It skips interposer setup for a static binary or a musl environment and delivers actual environment variables instead (see "Why Fork+Exec" above).

kubectl exec Threat Model

Admission and runtime bootstrap are separate gates

The operator denies API pods/exec into protected Pods by default, including kubectl exec. Kubernetes RBAC permission alone does not bypass this gate. The narrow cloudtaser.io/allow-exec exception requires an explicitly listed ServiceAccount and emits an admission warning; it does not disable eBPF.

Kubelet exec startup, readiness and liveness probes use the node's CRI path, outside API CONNECT admission. Current agents support this runtime bootstrap through kernel-observed executable identity and service lineage. They do not need an extra CAP_SYS_PTRACE grant or readable /host/proc/1/root for runtime enrollment. Bootstrap authority is revoked when the workload command execs.

Verify the workload's actual probe and the agent's runtime-enrollment and enforcement reports. Empty or degraded enrollment is an enforcement gap to diagnose, not a reason to disable protection or broaden runtime identity. An agent reporting ready alone does not prove every workload probe succeeds.

When runtime bootstrap and the operator's exec admission policy permit a command to start, kubectl exec -- env does not inherit the wrapped application's fetched secrets. The runtime creates the exec process as follows:

  1. kubectl exec first passes API CONNECT admission; the default protected-Pod policy denies it here.
  2. If authorized, the kubelet calls the CRI (Container Runtime Interface) Exec streaming RPC. Kubelet exec probes instead use ExecSync, without API CONNECT admission.
  3. The runtime creates a new process inside the container's namespaces. This new process is not a child of PID 1 (the wrapper) or PID 2+ (the application). It is spawned directly by the runtime
  4. The new process's environment is constructed from the container spec -- the same env fields visible in the pod YAML -- not inherited from any running process
  5. Since secrets are fetched by the wrapper after the container starts, the container spec does not contain them

When the exec command starts, kubectl exec -- env shows only the environment from the container spec (which contains CLOUDTASER_* variables -- OpenBao address, secret paths, env mapping -- not the secrets themselves). The wrapper strips CLOUDTASER_* vars from the child's inherited env, but that is a separate concern -- the kubectl exec process never inherits from the child at all.

Defense in depth

Active eBPF enforcement denies cross-process reads of the protected child's /proc/<child_pid>/environ and /proc/<child_pid>/mem. This protection is independent of API admission. A new exec command that reaches the container does not gain permission to inspect another protected process. Validate this layer with guest-local node/CRI streaming exec in an isolated test environment; an API admission denial is not a kernel secret-read proof.

Security Properties

Attack Vector Secret Visible? Notes
kubectl exec -- env No Denied by admission by default; an authorized runtime-created process receives the container spec environment, not the wrapped child's environment
kubectl exec -- cat /proc/<child>/environ No Denied by admission by default; healthy eBPF enforcement also denies the protected-child read from an exec that reaches the container
kubectl exec -- cat /proc/<child>/mem No Denied by admission by default; healthy eBPF enforcement also denies the protected-child read from an exec that reaches the container
kubectl cp of any file No Secrets are not on any filesystem
Core dump / crash dump Blocked by eBPF eBPF prevents core dump generation
Container runtime inspect No Secrets are not in container env spec
etcd / Kubernetes Secrets No Secrets never enter Kubernetes API
Node access + read /proc Blocked by eBPF eBPF blocks cross-process environ/mem reads
Cloud provider audit logs No Secrets transit only from EU OpenBao to pod memory
strace / ltrace on process Blocked by eBPF ptrace attach denied
Heap dump (jmap, gcore) Blocked by eBPF ptrace / process_vm_readv denied
Application logging env var values Yes Application-level responsibility

Trust Boundaries

  1. EU OpenBao is the trust root. Secrets originate only from EU-hosted OpenBao.
  2. Wrapper is trusted code. It runs as PID 1 and handles OpenBao authentication, secret fetching, and environment preparation.
  3. The application is untrusted. It receives secrets through the selected delivery mode. eBPF enforcement restricts reads of other protected processes; the application can still copy or expose secrets it is authorized to receive.
  4. The cloud provider is untrusted. Secrets never enter provider-controlled storage, APIs, or logging systems.
  5. The Kubernetes control plane is untrusted. Secrets never enter etcd, Kubernetes Secrets, ConfigMaps, or container env specs.

Secret Rotation Strategies

In direct-Vault mode, restart and sighup detect static KV-v2 changes automatically. Each renewal tick reads the configured data path and compares the response's metadata.version with the version last delivered successfully. The same response supplies the changed values, so a separate metadata lookup and a second fetch are unnecessary. This uses the existing data-read permission: policies do not need read on secret/metadata/<path>. An edit before the first periodic check is still detected, and failed application remains pending for retry.

Checks use the configured renewal interval with up to 25% per-wrapper jitter. A tick checks at most 16 paths with a fair cursor; a round has a five-second request budget and a 256 KiB limit per data response. This deliberately reads values on each check; unchanged values are zeroed and discarded. Propagation also depends on source availability and application restart/reload time. Permission, timeout, invalid-version, or size errors are reported without acknowledging an unseen update. Successful dynamic credential replacements survive failures on other paths and remain pending until successfully applied.

none skips automatic static checks and leaves the application at its startup generation. Dynamic lease maintenance continues. SIGHUP still explicitly requests a fetch, then applies the configured strategy. After a restart, the readiness probe runs for the supervisor lifetime rather than the short rotation transaction.

Broker/beacon mode: automatic KV-v2 version polling is not supported by the current encrypted application-read protocol. Its sealed payload carries the inner secret values without KV version metadata, and there is no authorized metadata operation. Use SIGHUP, a restart, or cloudtaser-cli target rotate to trigger a fetch in this mode. The wrapper does not fall back to a plaintext proxy or acquire a direct Secretstore credential to work around this boundary.

After a successful re-fetch:

  1. Wrapper fetches the current secret values from OpenBao
  2. Wrapper applies the configured strategy using the complete refreshed secret set
  3. Only successful application of that strategy acknowledges the fetched version
Strategy Behavior Use Case
restart (default) Wrapper sends SIGTERM to child, waits for exit, re-fork+execs with updated env Databases, services without durable application state, connection-pooling apps
sighup Wrapper updates the shared interposer data and sends SIGHUP; the child must re-read values through supported getenv() access. A runtime's startup environment snapshot does not update itself. Apps with compatible config reload support
none No child restart, signal or delivery-buffer update; the child keeps its startup values even when a newer store generation is detected Short-lived jobs, CronJobs

With restart, the child exits and the wrapper prepares a fresh delivery mapping or environment before launching the replacement. A snapshot runtime takes a new startup snapshot, provided its delivery mode populates that snapshot. Restarting default strict-mode Python alone does not make os.environ compatible: select snapshot strip mode first. Changes to the memfd or libc getenv() results do not update an existing Python dictionary, Go environment map, or application-level cached value.

Runtime Compatibility

Fork+exec support does not imply identical environment semantics. The table distinguishes the default with an active interposer from the static/musl fallback. Check the actual entrypoint, image, and secret-reading API, including any shell or subprocess in the startup path.

Runtime or access path Default behavior Required setup and rotation
Eligible glibc-dynamic code calling libc getenv() / secure_getenv() Mapped secrets are served by the interposer; absent from the exec environment Strict mode works for these calls. A reload must call the API again after rotation; cached application values do not refresh themselves.
CPython os.environ / os.getenv() Startup snapshot lacks the mapped keys, even when libc getenv() can return them Use CLOUDTASER_ENV_SNAPSHOT_MODE=strip and restart rotation.
Shell $VAR Builds with shell auto-selection use snapshot strip mode for sh, bash, dash, ash, ksh and zsh entrypoints, unless native-FD or inherited-env delivery is selected No extra snapshot variable on those builds. Older wrappers need explicit CLOUDTASER_ENV_SNAPSHOT_MODE=strip; use restart rotation and check each consuming process.
Perl %ENV Startup snapshot lacks mapped keys in strict mode Select snapshot strip mode explicitly and use restart rotation.
Statically linked Go (CGO_ENABLED=0), static C/Rust, or a musl environment Wrapper selects actual exec-environment delivery instead of interposer delivery Native environment access receives inherited values. Descendants can inherit them too; restart refreshes startup state. Go receives the exec environment, not a required read of /proc/self/environ.
Dynamically linked Go using os.Getenv Go still uses its environment snapshot; enabling CGO alone does not route this API through libc getenv() Use CLOUDTASER_KEEP_SECRETS_IN_ENV=true and restart rotation. Snapshot strip mode does not work for this API: its libc setenv() calls do not update Go's copied envp.
Node.js, Ruby, PHP, JVM, .NET, and other language APIs Depends on whether the particular API calls interposed libc functions or reads a runtime/application snapshot Verify the API in the deployed image. Use strict mode only for compatible libc reads; snapshots need explicit population and restart. A language name alone is not a compatibility guarantee.

CLOUDTASER_KEEP_SECRETS_IN_ENV=true is an alternative when the application needs real exec-environment inheritance, including children that lose the interposer. It expands plaintext exposure to the child and its descendants. Snapshot strip mode instead removes mapped keys at interposed exec boundaries; it does not erase the parent's snapshot, cover raw exec syscalls, or guarantee that a descendant can obtain secrets. Keep eBPF enforcement active and use the configuration reference to assess the complete process tree.

Python setup and compatibility check

For a glibc-based CPython application, add snapshot strip mode to the existing injected workload's Pod template and use restart rotation. Keep its existing TLS secret-store configuration, authorized secret mapping, and eBPF enforcement. This fragment contains only public settings; secret values still come from the configured secret store:

metadata:
  annotations:
    cloudtaser.io/rotation: "restart"
spec:
  containers:
    - name: app
      command: ["python3", "/app/main.py"]
      env:
        - name: CLOUDTASER_ENV_SNAPSHOT_MODE
          value: "strip"

Use a generated test credential mapped to APP_CREDENTIAL and run the following script as the wrapped application command in a test workload with the same image and configuration. A separate kubectl exec process does not exercise that delivery path. The check prints only presence booleans and exits nonzero if either access path lacks the key:

import ctypes
import json
import os
import sys

getenv = ctypes.CDLL(None).getenv
getenv.argtypes = [ctypes.c_char_p]
getenv.restype = ctypes.c_void_p
result = {
    "python_env_present": "APP_CREDENTIAL" in os.environ,
    "libc_getenv_present": bool(getenv(b"APP_CREDENTIAL")),
}
print(json.dumps(result))
sys.exit(0 if all(result.values()) else 1)

With an active interposer in default strict mode, Python can report false while libc reports true. With snapshot strip mode, both should report true. This check establishes presence for that workload; it does not prove rotation freshness or process isolation. After rotation, restart the Python child so its snapshot and any application caches are rebuilt from the refreshed secret set.