Where a step runs
Added in 2.4
Kraft's server schedules each step, but it does not start an agent or a command itself. It hands the step to a worker: a separate process, on the server's own computer or on another one you registered.
This page explains why it works that way, what travels each way, and how Kraft picks the place. To set one up, see Run a step on another machine. For every field, see workers.yaml.
One word needs care. Elsewhere in these docs a worker is an agent session. Here, the local worker and a machine are places a step runs, and neither is an agent session.
Every step goes through a worker
A step has two places to run:
- The local worker is one process for each installed Kraft, on the server's computer. Kraft starts it, and starts another if it dies. It needs no entry in
workers.yaml. - A machine is another computer listed in
workers.yaml, withkraft,gitand the agent CLIs installed.
A step with no runner: runs on the local worker. The server itself runs only builtin and forge tasks, such as making the worktree, opening the merge request and merging. Every agent session and every test command goes to a worker.
The local worker exists so a step does not depend on the server staying up. Restart the server and the steps of worktree items that the local worker holds carry on, and the next server picks them up. A machine gives the same separation across a network, and lets a step use a computer other than the server's.
The two differ in where the files are. The local worker works in the item's own worktree. A machine works in a copy of it, so something has to carry the work there and back.
What travels
A machine never talks to your forge. The server sends it everything a step needs and takes back what the step made. So a machine needs no forge credentials, and the server stays the only place that pushes.
- 1server to machineasks which of the item's commits the machine already has
- 2server to machinesends the missing commits as a git bundle, the files the step may read and write, and the jobThe machine runs the repo's setup command, then the agent or command, in its own copy.
- 3machine to serverstreams the log as the step writes it
- 4machine to serversends back its new commits as a bundle, and the files that changedThe server checks both, then moves the item's branch. Only then does it look at the step.
Out. The machine reports which commits it already holds, and the server sends the rest of the item's branch as a git bundle. Each item has its own copy on the machine, so an item's first step there receives its whole branch, and later steps only the new commits.
With the bundle go the files the step may read and write: the item's docs/ folder and earlier sessions' results and summaries. At most 5,000 files or 256 MiB go each way. In a repo that uses Git LFS, the objects the checkout needs travel beside the bundle too. The job says what to run, with its environment, git identity, time limits and the repo's setup command.
Back. The log streams home as it is written. After the step come the new commits, again as a bundle, and the files the step changed in the places it may write.
Never. The database, the gates and every decision on them stay on the server. So do the pushes and merges to the forge, and its credentials. A machine is never asked to push or merge.
No secrets. A machine comes set up: you install kraft and the agent CLIs there and log them in. The agent runs on the machine's own login, so no API key or token leaves the server. The env: values in repos.yaml do travel, so a secret does not belong there.
The machine looks up env_passthrough: names in its own environment. A sandboxed item sends declarations of its credentials, never their values.
A step's own calls to Kraft, such as kraft item progress or a reply to a review thread, cross the same connection. The server answers only what a worker session may call, and only for its own item.
What the server checks
The machine is trusted to run the step, not to describe its result. Everything it sends back is checked before the server takes it:
- Commits. The bundle must hold the item's branch and nothing else, and the server checks its objects as it fetches them. It must continue from the head the server sent. It may not add or change a submodule pointer (a gitlink), or add a nested repository.
- The branch. The server moves it only by compare-and-swap: only if it still points where the server expects. If the branch moved meanwhile, or the worktree has uncommitted changes to tracked files, the commits are kept under
refs/kraft/unsynced/and the item recordssandbox_branch_not_synced. Nothing is overwritten. - LFS objects. Each returned object must hash to an object a new pointer names, at that pointer's size, and every new pointer needs its object, sent back or already on the server. One that fails refuses the whole result.
- Files. The server writes files back only inside the places the step was given to write, never through a link. A count and size limit applies to what returns, and a refusal writes nothing. In
docs/, a file you changed while the step ran keeps your version. - Calls and events. A call outside a worker session's rights gets a 403. The machine may add only a fixed set of sandbox events to the item's history.
A result the server does not take fails the step, with the reason at the end of its log. The branch is left as it was.
How Kraft picks where
A step's runner: says where. An override on one node of the item beats an item-wide override, which beats the library's runner: (step, then node, then chain). The value is one of these:
| Value | Meaning |
|---|---|
local | The local worker. This is the default. |
| A machine's name | That machine, and no other. |
any | The first remote machine, in name order, that is online, may work on the repo and has a free slot. Never the local worker. |
| A group's name | Like any, but only the group's members. The local worker may be one, and is tried last. |
A machine group is a name you give several machines, so a step can ask for "a GPU machine" without choosing which. Machines and groups share one namespace.
Resuming goes back
A step resumed with the agent CLI's --resume, such as a paused session or an escalation thread's later turn, goes back to where it started, because the CLI's resume state lives there. A step placed on any also stays on the machine its node first ran on, for the rest of that pass of the node. Every other launch is placed afresh, so a fix-loop relaunch on a group can land on another of its machines.
Capacity is checked on both sides
Both ends set how many steps a machine runs at once. The server's capacity in workers.yaml and the machine's own capacity in machine.yaml both apply, and the lower wins. A machine that sets neither runs one step.
The server takes a slot when it chooses, so two steps never both start on the last free slot. Then it asks the machine to hold one. A machine at its own limit refuses, before anything is built or sent, and Kraft tries the next candidate or waits. That is not a failure or a retry, and the machine is left alone for 30 seconds unless it frees a slot sooner.
The machine keeps a limit of its own because the server cannot see what else runs on it. See The machine's own limit.
Waiting, and refusals
A step whose machine is offline, full or disabled waits. The board says why, as waiting for gpu-box (full) or waiting for a machine in gpu. The wait holds no slot and does not count against the step's time cap. Kraft never runs the step anywhere else to end the wait.
A placement that can never hold is refused instead. Kraft answers with a 422 when you file or override an item, and stops the step with a config_error when it launches. The causes:
- the machine or group is not in
workers.yaml; - the machine is limited by
repos:to repos that leave out the item's; - no member of the group could take the item;
- the item is one a machine cannot take;
- the machine's Kraft is too old for something the step needs.
A machine can also refuse at launch, when it lacks the agent CLI or cannot build the item's sandbox. See What a machine refuses.
Two ways to connect
A machine connects to the server in one of two ways, and everything above is the same over either.
- Dial-in. The machine opens the connection with a token, so it works behind a home or office router (NAT), and the server needs no route to it. The server must be reachable from the machine.
- Ssh. The server dials the machine with its own
ssh. A small background process on the machine keeps the step's state, so a step outlives the ssh session that started it.
The side that dials redials after a drop. Kraft checks that the server and the machine run the same major version. A machine on another major is turned away, and says why.
When a machine or the server goes away
A step keeps running when the connection drops: the laptop sleeps, ssh ends, the server restarts. The agent's calls to Kraft wait for the connection, and a call already sent fails with a 503 rather than risk acting twice. A step that finishes meanwhile keeps its log, commits and files on the machine, and hands them in on reconnect.
Neither side waits forever. give_up_after, 2 hours unless you set it, is how long either side waits:
- The machine stops the step once it has had no server for that long.
- The server ends the step as
unknown, and throws away what the machine hands in later. - Only time the server was awake counts, so a night's sleep does not use it up.
give_up_after can be never. The step then runs to its end and waits to hand in.
A sleeping laptop enforces nothing, so a step's time cap and idle-output limit are also enforced by the machine. Whichever side fires first stops the step. The local worker has no give_up_after, since it sleeps with the server. While only the server is down, nothing enforces a local step's caps; see Restart Kraft.
What a machine cannot take
A machine gets a bundle of one repository's branch. Two kinds of item do not fit in that, and stay on the local worker:
- An in-place item. It runs in your own checkout of the repo, which a machine does not have.
- An item with workspace members. A bundle carries one repo, not the members' repos.
Kraft refuses such an item for a machine, for any, or for a group without the local worker, when you file it. A group that includes the local worker takes it there instead. What a machine refuses has the exact rules.
A repo with submodules still runs on a machine when the item has no workspace members. Its submodules stay uninitialized there, as they do in the server's worktree, and Kraft refuses commits that move a submodule pointer.
A repo that uses Git LFS runs only on a machine that has git-lfs, and any and groups pass over the others. See Git LFS repos on a machine.
Related
- Then do it: Run a step on another machine.
- workers.yaml: every field, the connection modes, and each refusal.
- Admin verbs:
kraft admin worker. - Workers settings: the same machines in the web UI.
- Vocabulary: machine, local worker and machine group.