workers.yaml

Added in 2.4

workers.yaml lists the machines Kraft may run steps on. A machine is another computer with the agent CLIs installed and kraft on its PATH. A connected machine runs the steps that work items place on it.

workers:
  gpu-box:
    ssh: gpu-box              # ssh mode: a host or an ~/.ssh/config alias
    capacity: 2               # sessions at once, at most; unset: the machine's own limit, else 1
    repos: [kraft, infra]     # optional; left out means every connected repo
    give_up_after: 2h         # default 2h; "never" turns the limit off
    groups: [gpu, linux]      # optional; the groups a step's runner can name
    enabled: true             # default true; false: running steps finish, nothing new starts
  home-server:
    token_sha256: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
local:
  capacity: 3                 # sessions this machine's own worker runs at once; unset: no limit
  groups: [linux]             # optional; the local worker joins groups too
  enabled: true               # default true; false: local steps wait

kraft admin worker add and edit write this file for you. The file does not exist until the first machine is added.

Each machine sets exactly one of ssh and token_sha256.

The top-level local: key is not a machine. It holds the settings of the local worker: capacity, groups and enabled. Adding, editing or removing a machine keeps it.

The local worker

Starting Kraft also starts the local worker, a second process on the same machine that runs every local step: agent sessions and the commands of test tasks. It needs no entry in workers.yaml, no token and no worker add. The server finds it again after a restart, and starts a new one if it dies.

• It appears as local in Settings › Workers and in kraft admin worker list, with whether it is up, how many steps it runs and its Kraft version. It runs as many steps as local: capacity allows, and has no limit when that is unset; max_concurrent limits work items, not steps. It cannot be removed, and enabled: false makes local steps wait (waiting for local (disabled)) while the ones running finish.
• It listens on a unix socket under run/local-worker/ (a private folder under /tmp when that path is too long for a socket), and only your user can reach it. Its log is run/logs/local-worker.log.
• There is one worker for each installed Kraft, named by its version and a hash of where it is installed (worker-<version>-<hash>.sock). After an update the new server starts a worker of its own and takes over the running steps; the old one gets no new steps, reaps the processes it started and exits.
• Stopping or restarting the server does not stop the worker. The steps of worktree items it runs carry on, and the next server picks them up; kraft admin stop --now stops them. Under a systemd service the worker starts outside the unit's cgroup, so stopping the unit leaves it running.
• A step placed on local, or on nothing, waits while the worker is not connected, and the board says waiting for local. It is never run in the server instead.
• A step beyond local: capacity waits before it has a session. The board says waiting for local (2 of 2 running), the wait does not count against its time cap, and the step's log starts with how long it waited. Every local step counts, test commands included, so a low limit runs parallel test scopes one at a time. Steps the worker kept running through a restart count too, and a new one waits for them.
• A step the worker started keeps running if the worker dies. The next worker takes it over, and the server goes on watching it.
• It exits by itself a few seconds after the server is gone and nothing is running.
• A worker on another machine and the server must run the same major version of Kraft. The handshake turns a machine on another major away with update kraft on this machine to X.y; one on an older release of the same major keeps getting steps.

The machine's own limit

A machine can set its own limit, so it never takes more than it can run whatever a server sends. Put it in machine.yaml in the config directory of the Kraft that runs the machine's steps ($KRAFT_HOME/config):

capacity: 2

• The machine reports the limit when it connects and again when the file changes. It rereads the file before each step and every 20 seconds, so a raised limit reaches the server within 20 seconds. A file that does not parse is logged and read as no limit.
• The server uses the lower of its capacity and the machine's, and kraft admin worker list and Settings › Workers say when the machine's is the lower. A machine that sets none, or an older one, is limited by the server's capacity alone.
• Before it builds a step for a machine that has a limit, the server asks the machine to hold a slot. A machine at its limit refuses, and the step goes to another free machine it may use (any), or waits for the named one. Nothing has been made for it yet, so a refusal leaves no session and sends no files.
• A refusal does not count as a failure or a retry. The machine is not asked again until it reports a new limit, one of its steps ends, or 30 seconds pass.
• The limit belongs to the connection to one server. A machine used by several servers applies it to each; it is not shared between them.
• machine.yaml is kept across a major update.

Groups

A group is a name you choose for a set of machines. A step whose runner: names the group goes to one of its members, so you can say "a GPU machine" without picking which. A machine lists its groups in groups:, and can be in several. The local worker joins groups with local: groups:, and kraft admin worker add NAME --group G (repeatable) writes a machine's; edit NAME --group G replaces them.

Kraft takes the first member that is online, may work on every repo of the item, and has a free slot: remote members first, in name order, then the local worker. So a group that includes the local worker sends work to its machines first; an unlimited local worker would otherwise take every step. any is a separate word: every remote machine, never the local worker.

• The slot is taken when Kraft chooses, so two steps never both start on the one free slot, and a step is never held for one member while another is free.
• A member at its own limit turns the step away, and Kraft goes on to the next member. See The machine's own limit.
• With no member free, the board says waiting for a machine in gpu. The step starts on the first member that frees, looked at again every 30 seconds. A group that loses its last member while steps wait for it refuses them at the next check.
• A resumed session (a paused agent step, an escalation thread's later turn or a review reply) goes back to the machine it started on (or the local worker) and waits for it. Every other launch is placed afresh: a fix-loop relaunch, each test scope and a fallback harness each take whichever member is free.
• A remote member cannot take an in-place item or one with workspace members. Such an item runs on the local worker when it is a member. When it is not, Kraft refuses the item when it is filed, naming the group.
• A step that needs a capability a member lacks, a codex step needing Kraft's permission hook or a step of a Git LFS repo on a member without git-lfs, goes to a member that has it, else the local worker when it is a member.

Disabling a machine

kraft admin worker disable NAME (or enabled: false) is for taking a machine out of rotation without forgetting it: it keeps its token or ssh entry, its slots and its connection.

• Steps already running on it finish, hand in their results and give their slots back.
• Nothing new starts on it. A step that names the machine waits, and the board says waiting for gpu-box (disabled), even when the machine is also offline. A group or any goes on to another member.
• enable undoes it; the waiting steps start on the next check.
• kraft admin worker list shows disabled, with · offline: and the reason after it when the machine is also out of reach. kraft admin doctor reports it as disabled, not as a warning.

Names

A machine's name is 1 to 39 lowercase letters, digits, - or _, starting with a letter or digit. A group's name follows the same rule. Machines and groups share one namespace: a group named like a machine, or a machine like a group, is refused. local and any are refused for both: they are placement words.

Naming a repo

An entry in repos matches a connected repo by its id, its name, or the last part of its path. A name that matches no connected repo is refused when you save the file in Settings or add the machine with worker add. A hand edit that names one is shown as a problem on the pending reload.

Two ways to connect

Dial-in

The machine opens a connection to the server, so it works behind a NAT and needs no inbound port. This is the default mode: kraft admin worker add NAME with no --ssh registers one.

1. Run kraft admin worker add home-server. It prints a token once, and the command to run on the machine.
2. On the machine, run kraft admin worker serve --server URL --token TOKEN. It redials by itself after a drop, a refusal or an unreachable server, until you stop it.

The server must be reachable from the machine. That means binding it beyond loopback with a password, as in Remote access.

The connection route is /api/workers/connect. It is exempt from the login cookie: the machine proves itself with its token and nothing else. It still obeys the rest of the perimeter, so a server with no password refuses a peer that is not local, token or not.

Kraft speaks plain WebSocket, so TLS is your reverse proxy's job. Put the https:// address in --server.

The token appears in the process list on the machine, because it is on the command line. Treat that machine as trusted with it. To change it, run kraft admin worker rotate-token NAME: it prints a new token once, the old one stops working at once and the machine's connection ends. Steps running on the machine wait for it, up to its give_up_after, until serve runs again with the new token. An ssh machine has no token to rotate.

Ssh

The server dials the machine with ssh HOST kraft admin worker serve --stdio and redials when the connection ends. Nothing runs on the machine until the server connects.

Kraft uses the server process's own ssh and its own ~/.ssh/config. A server started by launchd or systemd may not reach your SSH agent, so give it a key file with no passphrase prompt, or an agent socket it can see. The connection runs with BatchMode=yes, so it never asks for a password. kraft must be on the remote machine's non-interactive PATH.

kraft admin worker add NAME --ssh HOST saves the machine, connects once and prints the result. A failed connect leaves the machine saved and exits 1, because a machine that is asleep or not yet reachable is still worth registering.

A machine that connects over ssh keeps its state in a small background process, the machine daemon, so a step outlives the ssh session that started it. The first connection starts it, detached, and later ones reuse it. It listens on a unix socket in $KRAFT_HOME/remote, which only the machine's user can enter, and logs to $KRAFT_HOME/remote/daemon.log.

The daemon exits after ten minutes with no connection and no unfinished step. After an upgrade, the first connection from the newer Kraft retires the older daemon as soon as it holds no step. A dial-in machine does not use it: serve --server is already a long-running process.

When a machine is offline

kraft admin worker list shows offline: and the reason.

kraft admin doctor shows each offline machine as a warning with the same reason. For an ssh machine it is the server's own attempt, not your shell's, so a failure there that your terminal does not show points to the SSH agent.

A second connection from a machine replaces the first. A machine that is removed, or whose ssh or token_sha256 changes, loses its connection at once.

How Kraft reads the file

• Kraft reads the file at startup and again on kraft admin reload. A hand edit shows in Settings as waiting to be reloaded.
• A file that does not load is not used: at startup no machine is connected, and on a reload the running set is kept. kraft admin doctor names the problem.
• kraft admin worker add, edit, enable, disable, rotate-token and remove write the file themselves and apply it at once.
• A Kraft session cannot add, edit, rotate, list or remove a machine. Only a person, or an assistant a person is working with, can.
• A new give_up_after is used by the server at once and reaches the machine the next time it connects.

When the server is gone

A step keeps running when the server goes away: the laptop sleeps, ssh drops, the server restarts. The agent's calls to Kraft wait for the connection and are answered after it comes back. A call already sent when the connection dropped fails with a 503 saying so, because sending it again could act twice.

A step that finishes while the connection is down keeps its log, commits and documents on the machine and hands them in when it reconnects. Its log carries on from the byte the server has.

give_up_after is how long either side waits:

• The machine stops the step it is running once it has had no connection for give_up_after, and writes kraft: stopped on this machine after 2h without Kraft at the end of the step's log.
• The server ends the step as unknown, the status a process that vanished gets, once the machine has been out of reach for give_up_after. It records remote_step_given_up, and tells the machine to stop the step if it comes back. Whatever the machine hands in after that is thrown away.
• Only time the server was awake counts. A laptop that sleeps for a night does not give up on a machine: the clock starts again on wake.
• never turns both off for that machine. Its steps run to the end and wait, however long, to hand in. The board shows the step as unreachable since the time the connection dropped, until the machine returns or you cancel the step.

A machine that holds a step cannot be removed: the server answers 409, naming the items, and says to disable the machine and remove it once the steps end. A step held by a machine that is offline is freed by pausing or cancelling its item, or by give_up_after. Steps only waiting for the machine do not block its removal; they are refused at their next check. The local worker cannot be removed at all.

The time cap and the idle-output limit of a step are enforced on the machine as well as on the server, since a sleeping laptop enforces nothing. Whichever side fires first stops the step with the same time_cap_reached event.

A machine drops its copy of an item, its working files and its job folders, when the item is archived or abandoned, and on reconnect for every item the server no longer lists. It keeps the copy of an item while one of the item's steps is still running there.

Running a step on a machine

Name a machine in a chain's runner: (see Placing a step on a machine) or give one item its own with kraft item create --runner NAME. A step placed there runs on the machine; the board, gates, repos and the item's branch stay on the server.

Before every step the server sends the commits the machine lacks as a git bundle, the item's docs/ folder, earlier sessions' results and summaries, and the job. The machine runs the repo's setup_command in its own copy first.

The log streams home as it is written. After the step the machine's new commits, result.json, summary.md and changed documents come back, and only then does the server look at the step.

The machine uses its own login for the agent CLI. No credential or environment variable leaves the server: env: values from repos.yaml travel, and env_passthrough: names are read from the machine's own environment.

A step's own calls to Kraft (kraft item progress, a reply to a review thread, a read of its own item) cross the same connection. The server answers only what a worker session may call, for its own work item.

An unsandboxed step has no session MCP endpoint (/mcp) on a machine, so it uses the kraft command. An escalation turn, a review reply or a gate review that runs on a machine gets the same worker rights and nothing wider.

Sandboxed items on a machine

A step of a sandboxed item runs on a machine inside the sandbox the item was filed with, built by the machine. The server sends the sandbox value, the network lists and the credential declarations, never a secret: the secrets are read from the machine's own environment. A machine that cannot build the sandbox stops the step as config_error, naming itself and what is missing, and runs nothing. See On a remote machine.

kraft admin worker check shows one sandbox: row per backend, docker and os, with its own health detail, and skips a backend that does not work there. When none works it warns that sandboxed items cannot run on the machine.

A sandboxed worker's calls to Kraft, its permission requests over /mcp included, go out through the machine's egress proxy and across the connection. The machine refuses a call outside the worker's allowlist before it sends it, and the server checks it again.

A sandboxed codex step whose policy needs its permission hook cannot run on a machine.

Codex steps on a machine

Codex skips a hook it does not trust and says nothing. When a step's tool policy needs Kraft's permission hook, the machine's own codex trusts the hook, in the machine's copy of the worktree, just before the step starts. The server's codex is never asked, because another version may hash the hook another way. The hook runs with the machine's own Kraft.

If the machine's codex cannot trust the hook, the step stops as config_error with the reason in its log and runs nothing. A machine whose Kraft is too old to do this check is refused the same way, with update Kraft on that machine; its other steps still run.

Git LFS repos on a machine

A step of an item in a Git LFS repo runs on a machine that has git-lfs. A machine without it is never given the step: any and a group pass it over, and a named machine is refused with a message that starts the repo uses Git LFS: machine 'gpu-box' has no git-lfs; install git-lfs there.

kraft admin worker check shows a git-lfs row, skipped when it is missing. Kraft on the machine must be as new as the server's, or it does not list git-lfs.

The machine's worker looks for git-lfs once, when it starts, and keeps the answer. If you install it later, the machine does not list it until its worker restarts: kraft admin worker serve for a dial-in machine, and for an ssh machine the background process the server's ssh talks to.

An ssh reconnect does not restart that process while it runs; it exits on its own after about ten minutes with no job and no connection, and the next connection starts a fresh one.

The machine never fetches from the forge. The objects the step's head checks out travel through the server, beside the git bundle, one at a time. The server sends only those it holds, and only those the machine does not already have. Each item has its own copy on the machine, so an item's first step there receives every LFS object its head checks out: transfer time and disk, per item.

An object the server lacks stays a pointer file on the machine, as it would in a local worktree that could not fetch it.

The files the agent adds or changes come back the same way. Kraft checks each returned object against a pointer in the commits it came with, by hash and size, and refuses the whole result if one does not match, or if a new pointer has no object behind it. The branch then does not move.

What passes goes into the server's LFS store and is pushed with the branch, as a local step's would be. An object is sent again whole if the connection drops mid-transfer.

Waiting and capacity

A step whose machine is offline or full waits: the item shows waiting for gpu-box (full) (waiting for a machine in gpu for a group), holds no slot on the server, and its time cap does not run.

It never runs anywhere else. any means the first connected machine that may work on every repo the item uses and has a free slot; it never means the server. A machine runs at most as many sessions as its limit. Pause, skip and cancel work as usual.

A step that was resumed with the agent CLI's --resume (a paused session, an escalation thread's later turns) goes back to the machine that ran its first turn, because the CLI's resume state lives there. A step placed on any also stays on the machine its node first ran on, for the rest of that pass of the node.

What a machine refuses

Kraft refuses, with a 422 when the item is filed or overridden and config_error at launch, an item that names a machine or group not in workers.yaml, a machine not allowed for its repo, or a group none of whose members could take it; an in-place item; and an item with workspace members, which a single bundle does not carry.

A repo with submodules is not refused when the item declares no members: its submodules stay uninitialized on the machine as they do on the server, and Kraft refuses the machine's commits when they add or move a submodule pointer.

A machine without the agent CLI the step needs, or whose setup_command fails, stops the step with the reason in its log.

If something goes wrong

A bundle that does not fetch, adds a nested git repository, or does not continue from the head it was sent fails the step with the reason in its log and leaves the branch alone.

If the branch moved while the step ran, the commits are kept at refs/kraft/unsynced/... and sandbox_branch_not_synced is recorded. A document you edited while the step ran keeps your version; the step's copy is saved in the session's folder and remote_docs_conflict names it.

When the server restarts, a running step carries on and its log resumes without a gap.

Related

• Where a step runs: how the local worker and machines fit together, and why.
• Admin verbs: kraft admin worker add, edit, enable, disable, rotate-token, list, remove, serve and check.
• MCP tools: add_worker, edit_worker, rotate_worker_token, list_workers and remove_worker.
• Remote access: binding the server beyond loopback.
• Configuration: the other files in the config directory.