Caps, budgets and waits
The first column is the text the board and kraft view show ID print. ... stands for text that varies. A row either holds the whole fix or links to an entry below. Raise a cap on a stopped item covers lifting a cap and retrying.
Caps and budgets
| The reason says | What happened | Fix |
|---|---|---|
budget cap reached: ... on this work item | The item's own dollar cap refused the next launch. | Run kraft item raise-budget ID --usd N: it raises the cap and retries in one step. Or select Raise cap on the item. |
budget cap reached: ... today, across every work item | The daily cap, which nothing on the item raises, refused the next launch. | Raise budget.daily_usd in policy.yaml, or wait for local midnight, then run kraft item retry ID. raise-budget is refused, and the item's page offers Retry with no raise. |
budget_usd reached ..., token budget reached ... | A policy budget_usd or token_budget refused the next launch. | See A policy budget or token cap refused the launch. |
budget_usd cannot be checked: N launch(es) in ... reported no cost, and unknown spend is never counted as free ... | A launch ran on a harness that reports no cost, on a model Kraft can't price, so it can't show the scope is under its budget_usd. | Run kraft item set-policy ID --policy budget_usd=none. See A budget cap can't be checked. |
Time caps and waits
| The reason says | What happened | Fix |
|---|---|---|
<scope> hit its time cap of N minutes | A time cap ran out. | Run kraft item set-policy ID --policy time_cap_minutes=240, then kraft item retry ID. See Raise a cap on a stopped item. |
<scope> hit its total time cap of N minutes | A scope's wall-clock cap, which waits and gates count toward, ran out. | Run kraft item set-policy ID --policy total_time_cap_minutes=N, then kraft item retry ID. A node or task needs its path in the key, such as merge_request_feedback.ci.await_ci.total_time_cap_minutes=N. |
<task> showed no output for N minutes Added in 2.3 | An agent session wrote nothing to its log for agent_idle_minutes. | Read its log with kraft view logs ID, then run kraft item retry ID. If it was working, raise agent_idle_minutes in policy.yaml. |
gate <gate> waited past its timeout of N minutes | A gate went undecided past its timeout. | Run kraft item retry ID to reopen the gate, then decide it. |
<task> timed out waiting ... | An external wait (CI, a review) ran out. | Fix what the wait needed, then run kraft item retry ID. See Waiting on CI or a review. |
| A CI or automated-review failure, with a suggested retry | CI failed for its own reasons, or a cancelled run has no successor. | Run kraft item retry ID once CI has recovered. |
rate_limit retries exhausted after N attempt(s) | The agent's provider kept refusing for a rate limit. | Wait for the limit to reset, then run kraft item retry ID. |
A policy budget or token cap refused the launch
A policy budget_usd or token_budget refused the next launch. The message names the scope: the work item, or a node, step or task path in backticks.
raise-budget is refused for a token_budget or a path, and the item's page offers Retry with no raise. Raise a cap on a stopped item has the fix for each scope: Raise cap or kraft item raise-budget for the item-wide budget_usd, kraft item set-policy for a token_budget, and for a cap the chain sets on a path, kraft item skip or a change to the chain.
The item's policy was fixed when it was filed, so editing policy.yaml or the chain reaches only items filed afterwards.
A budget cap can't be checked
A launch in that scope ran on a harness that reports tokens but no cost, on a model prices.json can't price. Kraft can't show the scope is under its budget_usd and won't count the launch as free. Which harnesses these are is on Harnesses that report no cost. Raising budget_usd doesn't help: the stop stays while any budget_usd applies to the scope.
- Clear the item-wide cap with
kraft item set-policy ID --policy budget_usd=none. Add--policy token_budget=Nto keep a bound. - Run
kraft item retry ID.
The budget_usd=none fix clears a cap from the chain's own policy:, repos.yaml or a policy.yaml default too, but is refused under a maxima.work_item.budget_usd, and doesn't lift a cap the chain set on a node, step or task. For those, skip that node with kraft item skip, or bound that harness's scopes with token_budget in place of budget_usd and file the item again.
Waiting on CI or a review
Kraft keeps checking on both waits. One that outlasts its timeout stops the item with <task> timed out waiting. A step waiting for a machine never stops the item with timed out waiting.
merge_request_feedback waits on "no checks yet"
The merge request has no CI checks, so nothing ever passes. Do one of these:
- Add CI that runs on merge requests.
- Set
ci_checks: falseon the repo's entry inrepos.yaml, so both CI waits, before and after the merge, pass at once. - File the item on the
quick-taskchain.
A merge request that conflicts with its base branch doesn't wait here, although GitHub runs no checks on it. Kraft rebases the branch onto the base at once, and a conflict the rebase can't resolve stops the item instead of the wait running out.
external_approval keeps waiting
A branch rule on the forge requires an approving review. Have someone else approve the merge request, or remove the rule. Kraft checks again on its own.
On GitLab, which pipeline counts
Added in 2.2
Kraft reads the newest pipeline that ran for the branch's head commit, branch pipelines and merge request pipelines alike. A merged results pipeline counts too. It runs on a merge commit of the branch into its target, and Kraft accepts it when that merge commit was built from the current head.
Right after a push the log reads no pipeline for <sha> yet until GitLab starts a pipeline for the new head. A line after it naming a merged results pipeline says why that one wasn't taken: it tested another commit, or its merge commit couldn't be read.
Waiting for a machine
Added in 2.4
A step placed on a machine waits while that machine is offline, full or disabled. A step placed on a group, or on any, waits while none of its machines is free. Run kraft admin worker list: an offline machine shows why, and a full one shows its slots in use. The step starts when the machine is back or a slot frees, and Kraft looks again every 30 seconds.
| The item shows | What it means | Do this |
|---|---|---|
waiting for gpu-box (offline: ...) | The machine is offline, and the text says why. | Fix what it names. |
waiting for gpu-box (full) | Every slot of the machine is in use. | Wait for a slot. |
waiting for gpu-box (disabled) | Someone turned the machine's enabled off, so it takes no new steps. It wins over offline in the text, because it is a choice. | Run kraft admin worker enable gpu-box. |
waiting for a machine in gpu, waiting for any machine allowed for kraft | None of the group's or the allowed machines is free. | Wait: the step starts on the first member that frees. |
waiting for local (2 of 2 running) | A local step waits for a slot of the local worker, which local: capacity in workers.yaml limits. It has no session yet. | Wait for a slot, or raise local: capacity. |
waiting for local (disabled) | The same wait, with local: enabled: false. | Set local: enabled back to true. |
waiting for test group heavy: 36djfdgg is running it, or ... is running it on gpu-box | A test scope waits for another scope of its concurrency_group to finish on the same machine, the local worker or a remote one. | Wait: it starts when one ends. |
waiting for test group heavy is held on 2 machines | Several machines of the group are held. | Wait: it starts when one ends. |
To run the step elsewhere, change its placement with kraft item set-node-override ID --node N --runner local. Kraft never runs the step anywhere else, and the wait doesn't count against the step's time_cap_minutes; a total_time_cap_minutes still counts it. A step on a named machine or on any shows its wait on the board too, whatever the reason.
Launch and setup stops
Fix a work item that stopped before its task ran, because Kraft refused the launch, couldn't make the worktree, or found a repo setting, registration or model it can't use.
Forge stops
Fix a work item that stopped because the forge refused its credentials, rate-limited it, couldn't be reached, or no longer matches the account the item first used.