Run a step on another machine

Added in 2.4

A work item's steps run on the machine the Kraft server runs on. This guide moves one of them to another computer you already have: a build box, a GPU machine, a Linux host. The board, gates, repos and the item's branch stay where they are.

You need kraft on the other machine, the agent CLIs it should run installed and logged in there, and git. Kraft installs nothing and sends no credential. For a sandboxed item the machine also needs the sandbox runtime and the item's image.

Register the machine

In Settings › Workers, press Add machine; the dialog asks for the same things as the commands below, and shows a dial-in machine's token once. Or, on the server's machine, add it in a terminal. For one the server can reach over ssh:

kraft admin worker add gpu-box --ssh gpu-box --capacity 2

It saves the machine, connects once and prints connected, or why not. For a machine that can reach the server but not the other way round, leave out --ssh. The command prints a token once and the command to run on the machine:

kraft admin worker add home-server

Then run the printed kraft admin worker serve --server URL --token TOKEN on the machine. The machine must be able to reach the server, which means binding the server beyond loopback with a password; see Let another machine connect. See workers.yaml for the two ways to connect.

The machine can also limit itself. Put capacity: 2 in machine.yaml in its Kraft config directory, and it never runs more than two steps at once, whatever --capacity says on the server: the lower of the two is used. A step it turns away goes to another machine, or waits. See The machine's own limit.

Check the machine

On the machine itself, for a dial-in machine with the server's address and its token:

kraft admin worker check --server URL --token TOKEN

It lists git, each agent CLI with its login, one row per sandbox backend that works there and whether the server answers with a valid token. Without --server and --token it leaves out the server row, which is how to check an ssh machine.

It exits 1 on any failure; a machine with no usable sandbox backend only warns, because it can still run items that are not sandboxed. kraft admin worker list on the server shows whether the machine is online and why not.

Keep the machine's Kraft current

The machine and the server must run the same major version of Kraft. A machine on another major is turned away when it connects, with update kraft on this machine to X.y: it runs A.b, another major version, and kraft admin worker list shows that as why it is offline. A machine on an older release of the same major keeps getting steps.

The one exception is a codex step that needs Kraft's permission hook. A machine whose Kraft is too old to have its own codex trust the hook is refused that step; see Codex steps on a machine.

When you update the server to a new major version, update the machines too: kraft admin update on each.

Put a node on it

In a chain file, add runner: to a node:

nodes:
  - id: implementation
    runner: gpu-box

The value is a machine's name, a group's name, any (the first connected machine that may work on every repo of the item and has a free slot, never the server) or local. A step's own runner: beats its node's, and a node's beats its chain's. See Placing a step on a machine.

Put a step on a group of machines

A group is a name you give several machines, so a step can ask for "a GPU machine" without picking one. Put machines in groups when you add them, --group as often as you like:

kraft admin worker add gpu-1 --ssh gpu-1 --group gpu
kraft admin worker add gpu-2 --ssh gpu-2 --group gpu --group linux

Then name the group as the runner:, in a chain or with --runner gpu. The step goes to the first of the group's machines that is online, may work on the repo and has a free slot, and waits as waiting for a machine in gpu when none has. To let the local worker take steps too, join it in workers.yaml with local: groups: [gpu]. It is tried after the others. Group and machine names share one namespace. See Groups.

Place one item without editing a chain

kraft item create "tidy the parser" --runner gpu-box
kraft item set-overrides --runner gpu-box
kraft item set-node-override --node verify --runner local

The first files an item whose steps all run on gpu-box. The second changes it for the whole item. The third overrides one node and wins over both. A change takes effect at the next step launch, even on a node that has started.

Kraft refuses, with a 422, a machine or group that is not in workers.yaml, a machine not allowed for the repo, and an item that is in place or has workspace members. A group takes such an item only when the local worker is one of its members. A repo that uses Git LFS runs on a machine that has git-lfs; see Git LFS repos on a machine.

Read a waiting item

An item whose machine is offline or full shows waiting for gpu-box (full), or waiting for a machine in gpu for a group. It holds no slot on the server and its time cap does not run. It starts when the machine is back or a slot frees. Kraft never runs the step on the server instead. See Waiting for a machine.

What comes back

The log streams to the server as the step writes it, so kraft view logs -f works as for a local step, and pause and cancel reach the machine.

After the step, its new commits, result.json, summary.md and changed documents are taken in before the server looks at the step. The next step, or a gate, sees them wherever it runs. If the branch moved meanwhile, the commits are kept at refs/kraft/unsynced/... and the item records sandbox_branch_not_synced. See If something goes wrong.

When your laptop sleeps

When the laptop the Kraft server runs on sleeps, the step keeps running on the machine. When the laptop wakes, the connection comes back by itself: the server redials an ssh machine, and a dial-in machine redials the server. The log carries on where it stopped, and a step that finished meanwhile hands in its commits and documents.

Two things end a step early. Each machine has a give_up_after, 2 hours unless you set it. A machine that has had no connection for that long stops its step. The server, once that much of its own awake time has passed, marks the session unknown.

Only time the laptop was awake counts on the server's side, so a night's sleep does not use it up. Set --give-up-after never on worker add for a machine whose steps should wait however long it takes. The task pane then shows the step as unreachable since the time it dropped.

A step's time cap and idle-output limit are also checked on the machine, so a capped step stops on time with the lid closed and shows time_cap_reached when the laptop wakes. See When the server is gone for each rule, and Settings › Workers for whether each machine is reachable now.

Related

• Where a step runs: why a step runs through a worker, what travels to a machine, and what the server checks.
• workers.yaml: every field and connection mode.
• Admin verbs: kraft admin worker.
• Item verbs: --runner on create, set-overrides and set-node-override.