How workers and machines behave
This page is the run-time behaviour behind workers.yaml: what the local worker does, how a machine's limit and its groups decide where a step goes, and what happens when a machine or the server goes away. For why a step goes through a worker at all, see Where a step runs.
The local worker
Starting Kraft also starts the local worker, a second process on the same machine that runs every local step: agent sessions and the commands of test tasks. It needs no entry in workers.yaml, no token and no worker add. The server finds it again after a restart, and starts a new one if it dies.
- It appears as
localin Settings › Workers and inkraft admin worker list, with whether it is up, how many steps it runs and its Kraft version. It runs as many steps aslocal: capacityallows, and has no limit when that is unset;max_concurrentlimits work items, not steps. It can't be removed, andenabled: falsemakes local steps wait (waiting for local (disabled)) while the ones running finish. - It listens on a unix socket under
run/local-worker/(a private folder under/tmpwhen that path is too long for a socket), and only your user can reach it. Its log isrun/logs/local-worker.log. - There is one worker for each installed Kraft, named by its version and a hash of where it is installed (
worker-<version>-<hash>.sock). After an update the new server starts a worker of its own and takes over the running steps; the old one gets no new steps, reaps the processes it started and exits. - Stopping or restarting the server doesn't stop the worker. The steps of worktree items it runs carry on, and the next server picks them up;
kraft admin stop --nowstops them. Under a systemd service the worker starts outside the unit's cgroup, so stopping the unit leaves it running. - A step placed on
local, or on nothing, waits while the worker isn't connected, and the board sayswaiting for local. It is never run in the server instead. - A step beyond
local: capacitywaits before it has a session. The board sayswaiting for local (2 of 2 running), the wait doesn't count against its time cap, and the step's log starts with how long it waited. Every local step counts, test commands included, so a low limit runs parallel test scopes one at a time. Steps the worker kept running through a restart count too, and a new one waits for them. - A step the worker started keeps running if the worker dies. The next worker takes it over, and the server goes on watching it.
- It exits by itself a few seconds after the server is gone and nothing is running.
- A worker on another machine and the server must run the same major version of Kraft. The handshake turns a machine on another major away with
update kraft on this machine to X.y; one on an older release of the same major keeps getting steps.
Connecting a machine
The two modes, and what each value must be, are in Two ways to connect.
Dial-in. The machine opens the connection to the server, so it works behind a NAT and needs no inbound port. kraft admin worker add home-server prints a token once and the command to run on the machine, kraft admin worker serve --server URL --token TOKEN, which redials by itself after a drop, a refusal or an unreachable server until you stop it.
- The connection route is
/api/workers/connect. It is exempt from the login cookie: the machine proves itself with its token and nothing else. It still obeys the rest of the perimeter, so a server with no password refuses a peer that isn't local, token or not. - Kraft speaks plain WebSocket, so TLS is your reverse proxy's job. Put the
https://address in--server. - The token is on the command line, so it appears in the process list on the machine. Treat that machine as trusted with it.
- After
rotate-token, steps running on the machine wait for it, up to itsgive_up_after, untilserveruns again with the new token.
Ssh. The server dials the machine with ssh HOST kraft admin worker serve --stdio and redials when the connection ends. Nothing runs on the machine until the server connects.
- A server started by
launchdorsystemdmay not reach your SSH agent, so give it a key file with no passphrase prompt, or an agent socket it can see. - A failed connect from
worker addleaves the machine saved and exits 1, because a machine that is asleep or not yet reachable is still worth registering. - The machine keeps its state in a small background process, the machine daemon, so a step outlives the ssh session that started it. The first connection starts it, detached and later ones reuse it. It listens on a unix socket in
$KRAFT_HOME/remote, which only the machine's user can enter, and logs to$KRAFT_HOME/remote/daemon.log. - The daemon exits after ten minutes with no connection and no unfinished step. After an upgrade, the first connection from the newer Kraft retires the older daemon as soon as it holds no step. A dial-in machine doesn't use it:
serve --serveris already a long-running process.
The machine's own limit
A machine can set its own limit, so it never takes more than it can run whatever a server sends. Put it in machine.yaml in the config directory of the Kraft that runs the machine's steps ($KRAFT_HOME/config):
capacity: 2
- The machine reports the limit when it connects and again when the file changes. It rereads the file before each step and every 20 seconds, so a raised limit reaches the server within 20 seconds. A file that doesn't parse is logged and read as no limit.
- The server uses the lower of its
capacityand the machine's, andkraft admin worker listand Settings › Workers say when the machine's is the lower. A machine that sets none, or an older one, is limited by the server'scapacityalone. - Before it builds a step for a machine that has a limit, the server asks the machine to hold a slot. A machine at its limit refuses, and the step goes to another free machine it may use (
any), or waits for the named one. Nothing has been made for it yet, so a refusal leaves no session and sends no files. - A refusal doesn't count as a failure or a retry. The machine isn't asked again until it reports a new limit, one of its steps ends, or 30 seconds pass.
- The limit belongs to the connection to one server. A machine used by several servers applies it to each; it isn't shared between them.
machine.yamlis kept across a major update.
Groups
A group is a name you choose for a set of machines. A step whose runner: names the group goes to one of its members, so you can say "a GPU machine" without picking which. A machine lists its groups in groups:, and can be in several. The local worker joins groups with local: groups:, and kraft admin worker add NAME --group G (repeatable) writes a machine's; edit NAME --group G replaces them.
Kraft takes the first member that is online, may work on every repo of the item, and has a free slot: remote members first, in name order, then the local worker. A group that includes the local worker therefore sends work to its machines first; an unlimited local worker would otherwise take every step. any is a separate word: every remote machine, never the local worker.
- The slot is taken when Kraft chooses, so two steps never both start on the one free slot, and a step is never held for one member while another is free.
- A member at its own limit turns the step away, and Kraft goes on to the next member. See The machine's own limit.
- With no member free, the board says
waiting for a machine in gpu. The step starts on the first member that frees, looked at again every 30 seconds. A group that loses its last member while steps wait for it refuses them at the next check. - A resumed session (a paused agent step, an escalation thread's later turn or a review reply) goes back to the machine it started on (or the local worker) and waits for it. Every other launch is placed afresh: a fix-loop relaunch, each test scope and a fallback harness each take whichever member is free.
- A remote member can't take an in-place item or one with workspace members. Such an item runs on the local worker when it is a member. When it isn't, Kraft refuses the item when it is filed, naming the group.
- A step that needs a capability a member lacks, a
codexstep needing Kraft's permission hook or a step of a Git LFS repo on a member without git-lfs, goes to a member that has it, else the local worker when it is a member.
Disabling a machine
kraft admin worker disable NAME (or enabled: false) is for taking a machine out of rotation without forgetting it: it keeps its token or ssh entry, its slots and its connection.
- Steps already running on it finish, hand in their results and give their slots back.
- Nothing new starts on it. A step that names the machine waits, and the board says
waiting for gpu-box (disabled), even when the machine is also offline. A group oranygoes on to another member. enableundoes it; the waiting steps start on the next check.kraft admin worker listshowsdisabled, with· offline:and the reason after it when the machine is also out of reach.kraft admin doctorreports it as disabled, not as a warning.
When a machine is offline
kraft admin worker list shows offline: and the reason; each reason is in Offline reasons. For an ssh machine the reason is the server's own attempt, not your shell's, so a failure there that your terminal doesn't show points to the SSH agent.
A second connection from a machine replaces the first. A machine that is removed, or whose ssh or token_sha256 changes, loses its connection at once.
How Kraft reads the file
- Kraft reads the file at startup and again on
kraft admin reload. A hand edit shows in Settings as waiting to be reloaded. - A file that doesn't load isn't used: at startup no machine is connected, and on a reload the running set is kept.
kraft admin doctornames the problem. kraft admin worker add,edit,enable,disable,rotate-tokenandremovewrite the file themselves and apply it at once.- A Kraft session can't add, edit, rotate, list or remove a machine. Only a person, or an assistant a person is working with, can.
- A new
give_up_afteris used by the server at once and reaches the machine the next time it connects.
When the server is gone
A step that finishes while the connection is down keeps its log, commits and documents on the machine and hands them in when it reconnects. Its log carries on from the byte the server has.
give_up_after is how long either side waits:
- The machine stops the step it is running once it has had no connection for
give_up_after, and writeskraft: stopped on this machine after 2h without Kraftat the end of the step's log. - The server ends the step as
unknown, the status a process that vanished gets, once the machine has been out of reach forgive_up_after. It recordsremote_step_given_up, and tells the machine to stop the step if it comes back. Whatever the machine hands in after that is thrown away. - Only time the server was awake counts. A laptop that sleeps for a night doesn't give up on a machine: the clock starts again on wake.
neverturns both off for that machine. Its steps run to the end and wait, however long, to hand in. The board shows the step as unreachable since the time the connection dropped, until the machine returns or you cancel the step.
A machine that holds a step can't be removed: the server answers 409, naming the items, and says to disable the machine and remove it once the steps end. A step held by a machine that is offline is freed by pausing or cancelling its item, or by give_up_after. Steps only waiting for the machine don't block its removal; they are refused at their next check. The local worker can't be removed at all.
The time cap and the idle-output limit of a step are enforced on the machine as well as on the server, since a sleeping laptop enforces nothing. Whichever side fires first stops the step with the same time_cap_reached event.
A machine drops its copy of an item, its working files and its job folders, when the item is archived or abandoned, and on reconnect for every item the server no longer lists. It keeps the copy of an item while one of the item's steps is still running there.
Running a step on a machine
Name a machine in a chain's runner: (see Placing a step on a machine) or give one item its own with kraft item create --runner NAME. A step placed there runs on the machine; the board, gates, repos and the item's branch stay on the server.
An unsandboxed step has no session MCP endpoint (/mcp) on a machine, so it uses the kraft command. An escalation turn, a review reply or a gate review that runs on a machine gets the same worker rights and nothing wider.
Sandboxed items on a machine
A step of a sandboxed item runs on a machine inside the sandbox the item was filed with, built by the machine. The server sends the sandbox value, the network lists and the credential declarations, never a secret: the secrets are read from the machine's own environment. A machine that can't build the sandbox stops the step as config_error, naming itself and what is missing, and runs nothing. See On a remote machine.
kraft admin worker check shows one sandbox: row per backend, docker and os, with its own health detail, and skips a backend that doesn't work there. When none works it warns that sandboxed items can't run on the machine.
A sandboxed worker's calls to Kraft, its permission requests over /mcp included, go out through the machine's egress proxy and across the connection. The machine refuses a call outside the worker's allowlist before it sends it, and the server checks it again.
A sandboxed codex step whose policy needs its permission hook can't run on a machine.
Codex steps on a machine
Codex skips a hook it doesn't trust and says nothing. When a step's tool policy needs Kraft's permission hook, the machine's own codex trusts the hook, in the machine's copy of the worktree, just before the step starts. The server's codex is never asked, because another version may hash the hook another way. The hook runs with the machine's own Kraft.
If the machine's codex can't trust the hook, the step stops as config_error with the reason in its log and runs nothing. A machine whose Kraft is too old to do this check is refused the same way, with update Kraft on that machine; its other steps still run.
Git LFS repos on a machine
A step of an item in a Git LFS repo runs on a machine that has git-lfs. A machine without it is never given the step: any and a group pass it over, and a named machine is refused with a message that starts the repo uses Git LFS: machine 'gpu-box' has no git-lfs; install git-lfs there.
kraft admin worker check shows a git-lfs row, skipped when it is missing. Kraft on the machine must be as new as the server's, or it doesn't list git-lfs.
The machine's worker looks for git-lfs once, when it starts, and keeps the answer. If you install it later, the machine doesn't list it until its worker restarts: kraft admin worker serve for a dial-in machine, and for an ssh machine the background process the server's ssh talks to.
An ssh reconnect doesn't restart that process while it runs; it exits on its own after about ten minutes with no job and no connection, and the next connection starts a fresh one.
The machine never fetches from the forge. The objects the step's head checks out travel through the server, beside the git bundle, one at a time. The server sends only those it holds, and only those the machine doesn't already have. Each item has its own copy on the machine, so an item's first step there receives every LFS object its head checks out: transfer time and disk, per item.
An object the server lacks stays a pointer file on the machine, as it would in a local worktree that couldn't fetch it.
The files the agent adds or changes come back the same way. Kraft checks each returned object against a pointer in the commits it came with, by hash and size, and refuses the whole result if one doesn't match, or if a new pointer has no object behind it. The branch then doesn't move.
What passes goes into the server's LFS store and is pushed with the branch, as a local step's would be. An object is sent again whole if the connection drops mid-transfer.
Waiting and capacity
A waiting step holds no slot and never runs anywhere else: see Waiting, and refusals and Resuming goes back. Pause, skip and cancel work as usual on a waiting step.
What the server checks on a result
A machine is trusted to run a step, not to describe its result. Everything it sends back is checked before the server takes it:
- Commits. The bundle must hold the item's branch and nothing else, and the server checks its objects as it fetches them. It must continue from the head the server sent. It may not add or change a submodule pointer (a gitlink), or add a nested repository.
- The branch. The server moves it only by compare-and-swap: only if it still points where the server expects. If the branch moved meanwhile, or the worktree has uncommitted changes to tracked files, the commits are kept under
refs/kraft/unsynced/and the item recordssandbox_branch_not_synced. Nothing is overwritten. - LFS objects. Each returned object is checked against a pointer in the new commits: see Git LFS repos on a machine.
- Files. The server writes files back only inside the places the step was given to write, never through a link. At most 5,000 files or 256 MiB go each way, and a refusal writes nothing. In
docs/, a file you changed while the step ran keeps your version. - Calls and events. A call outside a worker session's rights gets a 403. The machine may add only a fixed set of sandbox events to the item's history.
A result the server doesn't take fails the step, with the reason at the end of its log. The branch is left as it was.
If something goes wrong
A result the server refuses is covered in What the server checks on a result.
A document you edited while the step ran keeps your version; the step's copy is saved in the session's folder and remote_docs_conflict names it.
When the server restarts, a running step carries on and its log resumes without a gap.
Related
- workers.yaml: every field, the connection modes, and each refusal.
- Where a step runs: why a step runs through a worker, and what travels.
- Run a step on another machine: set one up.