Merge queue and train drops
Added in 2.3
A GitLab merge train or GitHub merge queue doesn't merge a merge request when asked. It queues it, then runs a further pipeline on the merge request combined with the target branch and everything ahead of it. That pipeline can fail after the merge request's own pipeline went green, because the target moved or because it runs tests the merge request pipeline doesn't.
Detection. Kraft reads from the forge, each time the merge node runs,
whether the target branch merges through a train or queue. There is no setting
to turn on. A repository without one waits for the merge exactly as before.
- GitLab: the project must have both
merge_trains_enabledandmerge_pipelines_enabled. A project that doesn't report them, or a read the forge refuses (403 or 404), counts as having no train. - GitLab, queueing: on a train project Kraft adds the merge request through the
merge-train API with the reviewed commit's sha, so it never merges a head
nobody reviewed. Only when the project read says there is no train does Kraft
merge with
glab mr merge. - GitLab, a failed project read: a read that fails any other way, such as a 5xx,
a timeout or an answer that isn't the project, fails the merge as a failed
glab mr mergewould. On GitLab before 19.1glab mr mergecan merge without going through the train. - GitHub: the target branch has a merge queue. Kraft blames only a failed
merge-group run created after the merge request's latest enqueue, so a failure
from an earlier drop is never held against the current head. When there is
no such run, for example because the checks don't run on GitHub Actions, a
removal reason that names failed or timed-out checks or CI counts as
code, and that reason is what the repair is given.
A read of the train or queue that fails says nothing about the merge, so Kraft keeps waiting and notes it in the log. A drop is noticed on the next poll of the merge wait. When the forge hasn't yet written why (no GitLab system note, no GitHub removal event for this entry), Kraft keeps waiting and reads it again on the next poll, up to the merge wait's own timeout.
The drop. Each drop is recorded as a merge_queue_dropped event with its
cause, the forge's reason, the failed jobs and the head. See the
events. The cause is one of five:
| Cause | What it means | What Kraft does |
|---|---|---|
code | The train or queue jobs failed. | Rebases the branch onto the target as it is now, hands the failed jobs and their log tails to the repair, then restarts. |
conflict | The merge request no longer merges cleanly with the target. | Rebases the branch. A clean rebase restarts with no agent. A rebase that conflicts goes to the repair to resolve, then Kraft restarts. |
not_added | GitLab only: the merge request never reached the train, because its own checks failed while it waited to be added. | Treated as code. |
infra | The train or queue's own runners failed. | Asks the forge to queue the same head again, with no agent and no restart, up to 2 times. Then it stops for you. |
other | A person took it off the train or queue, or the forge's reason can't be read. | Stops for you. Kraft never re-queues what a person removed, and no repair, recovery or escalation agent runs for it. |
The restart. After the repair commits, or after a clean conflict rebase,
Kraft reopens the approved gates in the span, gives the span's fix loops a
fresh count and re-runs from restart_from. The head goes through the merge
request's pipeline and your final review again, and is then queued again.
The next agent is told why in a note. The review brief opens with a "Merge train/queue rejection" section giving the cause, the reason, the failed jobs and what the repair changed.
What stops the item. The repair must commit: a head the train already rejected would only be rejected again, so a repair that commits nothing, fails or asks a question stops for you.
When a repair had already committed before it
stopped, whether it asked, failed, was paused or ran out of budget, the item is
held at restart_from with the span's gates reopened. A commit counts on the
item's branch in its own checkout or in any workspace member still to merge,
not only in the one whose merge request was dropped.
A repair that stopped before committing leaves the item on the merge node.
Whatever brings it back there, a resume, the rate-limit retry or your
kraft item retry, runs the repair again, with your steer, and never queues
the head the train rejected.
Only a retry from the chain's chain_finalized gate (or, with none, the last
gate before the merge node), or from a node before it, goes through that gate
and then queues the head again. A retry from a node after it, a later gate
included, repairs again.
The same goes for a Kraft restart while the repair works or the restart is
under way: if a repair had started or the branch moved, the resume holds the
item there and stops for you. A retry then walks back through final_review,
and the commit is never merged unseen.
The cap. Rewinds are counted under <node>.on_merge_rejected: 3 attempts
over 7 days unless policy.yaml names that key under loops:. See
Loop keys. Kraft checks it before
the repair agent runs, and at the cap the item stops with the kind cap,
without running an agent. A retry from a person resets it.
Skipping a node. When kraft item create --skip-nodes skips the node that
restart_from names, it moves to the nearest earlier exec node that is kept. If
none is left, the policy is dropped.
Writing the node
The shipped default chain sets on_merge_rejected on its merge node:
- id: merge
kind: exec
tasks:
- id: merge
extends: merge_mr
on_merge_rejected:
restart_from: merge_request_feedback
repair:
tasks:
- id: repair
extends: repair_merge_queue_failure
The keys are in on_merge_rejected.
A node with on_merge_rejected can't also declare a fix_loop: only the repair answers a drop, because a fix cycle's commit would be merged by the next measure unseen.
It may declare an on_failure, on the node or on one of its own steps or tasks, for a merge that fails for another reason than a drop, but only a read_only one. Such a handler can wait, retry, comment or ask for approval again. Kraft then asks the forge to merge again in place, the head final_review approved.
- id: merge
kind: exec
tasks:
- id: merge
extends: merge_mr
on_failure:
read_only: true
tasks:
- id: wait
extends: await_external_approval
on_merge_rejected:
restart_from: merge_request_feedback
repair:
tasks:
- id: repair
extends: repair_merge_queue_failure
An on_base_changed on this node, or on any node with an mr.merge task after the chain_finalized gate, must restart before that gate: a conflict resolution would otherwise be merged unseen.
What the repair and the escalation do
A node's on_merge_rejected names the
repair and where the chain restarts. In the shipped chain the gates in the
restarted span include final_review, so you see the repaired code before it
goes back onto the train.
If a read_only handler changes the worktree all the same, the item stops with
read_only_violated, held at restart_from with the span's gates reopened, so
no retry merges the change unseen.
Escalation. A commit made while the item is stopped on this node is never
merged in place, whether its escalation task made it or an escalation agent
did, Kraft's automatic one or one you sent. Kraft holds the item at
restart_from with the span's gates reopened, the way it does after a
repair's commit.
A successful escalation task that committed rewinds the chain there, under
the same cap, instead of re-running the node; one that changed nothing re-runs
it. A tip Kraft can't read, before the escalation or after it, counts as a
change. An escalation agent's own kraft item retry of this node, or of any
node after restart_from, restarts at restart_from; one it names at or
before restart_from already passes the gates again, and runs as named.
Without on_merge_rejected, a drop stops the item at once, with the forge's
reason in the stop, even when the drop comes on the re-measure after the
node's on_failure. No on_failure or fix loop on the node runs for it, and
no escalation agent is sent to the stop: Kraft never re-queues a merge request
on its own.