Resource limits

A sandbox's resources key limits its container's CPU, memory and processes. This page lists the three fields and says what Kraft does when a limit cannot be enforced or a process is killed for memory.

repos:
  - path: /home/you/code/my-service
    sandbox:
      kind: docker
      image: ghcr.io/acme/agent:1
      resources: {cpu: 2, memory: 4g, pids: 512}
      network:
        runtime: {allow: [api.github.com]}

Memory and swap

Swap is held to the same limit (--memory-swap) wherever the runtime can limit swap. Where it cannot, the container may swap as much again, and doctor says memory (swap unbounded).

How the limits bind

The limits bind every sandboxed launch of the item, its setup command included, and they are part of the sandbox: a chain, node or task cannot loosen, tighten or drop them. resources: {} sets none, and is the same sandbox as no resources at all.

When a limit cannot be enforced

A limit is applied or the task does not start. Two runtimes would otherwise drop a limit silently:

• Docker runs a container without a limit whose cgroup controller is missing, with only a warning.
• Rootless Podman on cgroup v1 ignores every limit.

Kraft therefore asks the runtime what it can enforce (Docker's info, Podman's cgroup controllers). A task that sets a limit the runtime cannot enforce stops as a configuration error that names the limit.

The fix is cgroup v2 and, for a rootless runtime, delegating the controllers to your user (systemd Delegate=cpu memory pids for user@.service).

A runtime whose info gives no answer Kraft can read (a daemon still starting) enforces nothing as far as Kraft knows. A task that sets a limit stops, and the next launch asks again. The default pids is left out, never refused, where the runtime cannot limit processes.

kraft admin doctor names the limits the runtime enforces on each sandboxed repository's row, and fails a row whose limits it cannot.

Out of memory

A session container is kept after it exits, so Kraft can ask the runtime whether the memory limit killed it, then removes it by name.

• The container has no log of its own (--log-driver=none); Kraft reads the output as it runs.
• The docker run client runs in a directory of Kraft's under $KRAFT_HOME/run/sandbox-client/, never the worktree.

When a process in the sandbox was killed by its memory limit, Kraft records a sandbox_oom_killed event ({session_id, memory, confirmed}). The item stops and waits for you as a configuration error naming the limit: the same limit would kill a retry the same way, so no fix loop runs. A session that still reported a result of its own (something it ran was killed and it got past it) keeps that result, with the event recorded.

A kill is confirmed by the runtime's own word: Docker's out-of-memory flag, and on Podman also the oom file its conmon writes where the client runs, since Podman on cgroup v1 never sets the flag. The runtime can set its flag a moment after the container exits, so for a container that exited 137 under a memory limit Kraft asks again for up to two seconds.

Docker on cgroup v2 can drop the flag altogether (moby#41929). Kraft counts a container as an unconfirmed kill when all three hold:

• it exited 137 under a memory limit
• Kraft did not stop it itself (a pause, a time cap, a cancel)
• the flag is still missing after those two seconds

The event then has confirmed: false, and the stop says the runtime did not confirm it was the limit.

Exit 137 from a container with no memory limit is never counted. A setup command the limit killed stops the item the same way.

Related

• Sandboxed workers: what else reaches the container.
• sandbox.yaml: how Kraft asks the runtime what it can enforce.
• Callbacks from a sandbox: how a sandboxed worker reaches Kraft.