docs(ci): omni-queue runbook for the final fix wave, the rollback's last step and who is told

docs/ops/OMNI_QUEUE.md says what the code does after this wave:

- Rolling the code back: a step 6 for the bless path (the release rulesets stay and, with
  OMNI_QUEUE_ENABLED false, nothing can write the land status: delete the rulesets right after the
  merge, or set the variable back to true so the rolled-back queue runs); Rollback's first bullet
  points at it. The rollback table gains landingsWithoutVerdict, stalled.quietTicks and the
  change in what counts as work; its landWithdrawalsOverflow row agrees with step 1.
- Stored withdrawals (G1): queue work found through the open dashboard issue (do not close it),
  retried on every visit, a stall after an hour, the day alert, the count saved only with another
  change or the heartbeat, the kept candidate ref; the incident entry follows (no "dispatch once
  per retry" any more), and the known limit that said they were not visited is gone.
- Who is told, and when (M8): the new watchdog signals (labelled PRs with no state file, an
  invalid config), the dashboard rules, and what still tells nobody; the invalid-configuration
  incident; the deadline's stop between a comment and its update.
- Health: the at-once pauses (error, tree-divergence), the withdrawal and landings-without-a-verdict
  rules, the episode that failed ticks opened held for two completed ticks; The queue stalled lists
  the new reason lines.
- The candidate verdict (G4): only the jobs of the candidate's push runs count; a check run that is
  no job is ignored and noted; a later same-named run of another suite supersedes nothing.
- PR gates (G3 was in its own commit), `queue why` reading gates and lane, the pause reason (one
  line, 200 characters, `Kept-nowhere`), who may pause and resume (M9), the base-repair lane's infra
  failure under base-red, the deferred bisection segment under a pause, a base-red pause with
  overflow entries, the rate-limited squash-api merge (no longer a known limit), the
  workflow_dispatch sentence, the scheduled-run timing, and an entry for the messages that had none.

English only; nothing under docs/i18n. runbook.test.mjs, prettier, check:doc-links,
check:fabricated-docs and check:docs-all (PR mode) pass; no broken in-page anchor.
This commit is contained in:
diegosouzapw
2026-11-16 20:15:01 -03:00
parent 9929d2c60e
commit 359ecf8856

View File

@@ -212,15 +212,20 @@ land statuses of each stored commit, writes `failure` where the same rule finds
App's newest, or the newest of the App's and those without a readable creator), and drops the entry
once that write succeeds or neither is `success` any more (or there is no status: GitHub answers an
empty list for a commit it does not know, so such an entry leaves at its next retry). A refused
retry keeps the entry and counts an attempt, and so does a failed read: `attempts` is the number of
failed attempts so far (a read that failed or a write that was refused). No count drops an entry: a
tick that visits the base never gives up on withdrawing a blessing. But stored withdrawals do not
make a tick visit a base: a base whose only work is a stored withdrawal (no batch in flight, no
pause, no labeled PR) is not visited, and its entries wait (see the known limits in [Red
candidates](#red-candidates-classification-base-check-bisection) and [A withdrawal GitHub
refused](#a-withdrawal-github-refused-or-one-refused-for-a-day)). When an entry's `attempts` reach
288 (`WITHDRAWAL_ALERT_ATTEMPTS` in `scripts/merge-queue/tick.mjs`, about a day of ticks), one
`error` event names the commit and its list and says the queue keeps retrying. A read or a write
retry keeps the entry and counts an attempt, and so does a failed read. A tick whose only change is
that count does not save the state (the hourly heartbeat, or any other change of the same tick,
writes it), so `attempts` counts the failed retries of the ticks that saved, a lower bound. No count
drops an entry: the queue never gives up on withdrawing a blessing. A stored withdrawal is queue work
(`hasQueueWork` in `scripts/merge-queue/contracts.mjs`): a tick run without a base visits its base
through the base's open dashboard issue even when no labeled PR targets it (`discoverBases` in
`scripts/merge-queue/tick.mjs`; see [State](#state)), and the watchdog judges that base (see [Who is
told, and when](#who-is-told-and-when)). An entry first refused more than 60 minutes ago makes the
queue stalled (`STALL_WITHDRAWAL_MINUTES`, see [Health](#health)): one alert mentions the approvers.
An entry refused for a day (`WITHDRAWAL_ALERT_MINUTES` in `scripts/merge-queue/tick.mjs`, 1,440
minutes from its `since`) adds one `error` event that names the commit and its list and says the
queue keeps retrying. While an entry is stored, the candidate ref of its batch
(`omni-queue/<sanitized-base>/<batchId>-<sha12>`) is kept, so GitHub still knows the commit when a
later tick writes `failure` on it; once the entry is gone, the next cleanup deletes the ref. A read or a write
that fails with a rate limit stops the retries of that tick: the rest of both lists waits for the
next tick. A commit the tick blesses for landing, a commit that lands, or a commit an approver
blesses by hand leaves both lists. Shadow mode retries the stored withdrawals too, and withdraws the
@@ -294,7 +299,8 @@ In live mode:
error whose message says `rate limit`. The next tick looks at the base again, recognizes a landing
that did happen, or writes the status and lands again. Each such attempt is counted in
`inFlight.landAttempts` and saved before its blessing, so a tick that died after the blessing
counts too. After `LANDING_ATTEMPTS_LIMIT` (3, in `scripts/merge-queue/tick.mjs`) attempts without
counts too. It also counts in `landingsWithoutVerdict`, the base's run of landings without a
verdict whatever the candidate (see [Health](#health)). After `LANDING_ATTEMPTS_LIMIT` (3, in `scripts/merge-queue/tick.mjs`) attempts without
a verdict, counted since the candidate was composed or the queue was last paused or resumed by
hand (they can be days apart), the batch is not blessed again and the base is paused with reason
`error` (see [Landing paused after three attempts](#landing-paused-after-three-attempts)). The
@@ -303,14 +309,16 @@ In live mode:
recomposition, a new batch) and an operator's pause or resume start it again. Shadow mode never
checks the limit (it blesses nothing), so a count carried from live mode never pauses a shadow
tick.
- **A per-PR merge call of `squash-api` that fails**, a rate limit included, is not retried by
the next tick. The GitHub client retries a rate-limited call within its own limits (4 retries,
waits of at most 120 seconds); after that the error is caught inside `landSquashApi` and is a
verdict, `merge-failed`, so the base is paused with reason `error`. The token hint is decided
on the text of the pause detail: with `actor: "owner"` a detail that says `rate limit` never gets
`(check OMNI_QUEUE_LAND_TOKEN: landing.actor is "owner")`; any other 401 or 403 gets it, a rate
limit whose message lacks those words included (see the known limits in
[Red candidates](#red-candidates-classification-base-check-bisection)).
- **A per-PR merge call of `squash-api` that fails** is a verdict, `merge-failed`, and the base is
paused with reason `error`, except for a rate limit. The GitHub client retries a rate-limited
call within its own limits (4 retries, waits of at most 120 seconds); when they run out,
`landSquashApi` keeps the client's mark on its result, and the tick handles it as a landing
without a verdict (the item above): the status is withdrawn, the tick fails with
`landing <batchId> hit a rate limit (<detail>)`, nothing is paused, no token hint is added, and
the attempt counts. The PRs merged before the limit are on the base; the next tick finds the base
moved and recomposes the others. For any other refused merge, the token hint is decided on the
text of the pause detail: with `actor: "owner"` a detail that says `rate limit` never gets
`(check OMNI_QUEUE_LAND_TOKEN: landing.actor is "owner")`; any other 401 or 403 gets it.
The land token is fine-grained and valid for 90 days: write the date it expires next to the secret
and rotate it before then.
@@ -397,8 +405,8 @@ existed) reads `API Route Typecheck missing` and waits like any other gate that
push or marking the draft ready runs it. Nothing is done about it: the
label stays, no comment is posted, and the PR becomes eligible by itself once its gates turn green.
The reason is computed by every tick and shown on the dashboard only; it is never stored in the
state. `npm run queue -- why <PR>` does not read the PR gates, so it can print `ELIGIBLE` for a PR
the dashboard shows as `pr-checks-not-green`.
state. `npm run queue -- why <PR>` reads the PR gates the same way and prints each gate's state, so
it agrees with the dashboard.
How a gate is read, per head (the check runs of the PR's current head SHA):
@@ -487,11 +495,14 @@ lanes.
red repair) and composes a new repair batch. Under any other
pause reason (`manual`, `infra`, `error`, `tree-divergence`) nothing is processed, lanes
included. In live mode a repair that lands lifts the `base-red` pause: a `resumed` event with
the detail `base repaired by #<n>`. While `landWithdrawalsOverflow` holds entries the pause stays
(the resume rule, see [CLI](#cli)); the tick after the one that empties the overflow then lifts
it because the tip moved, recorded as
`base moved <sha12> → <sha12> after a base-red pause`. In shadow mode the repair would land
only, and the pause lifts when the real tip moves.
the detail `base repaired by #<n>`. A `base-red` pause loaded together with entries in
`landWithdrawalsOverflow` is replaced by the full-list pause before the lane runs, so no repair
is blessed under it. In shadow mode the repair would land only, and the pause lifts when the
real tip moves. A repair whose CI fails for infrastructure after its `infraRetries` retries
is dropped and the `base-red` pause stays as it was, with one `error` event
(`<batchId>: base repair #<n> failed for infrastructure after <k> retry(ies) (<detail>): dropped; the base-red pause stays`);
the PR keeps its labels, so the lane composes it again, as a new candidate, in the same tick or
a later one.
- **A repair that does not repair.** A repair candidate that fails like the base (classified
base-red) is recorded in the state field `repairTried` (PR number → head SHA), and that head is
never composed again. A base that was not paused yet is paused as `base-red`. In live mode the PR
@@ -521,7 +532,10 @@ queue deliberately reads none (see below):
- no other check run in `failure`, `cancelled`, `timed_out`,
`action_required`, `startup_failure` or `stale`, except the names in `advisoryChecks`
(`Build (advisory)`);
- nothing pending. Only the latest run per check name counts (re-runs report several);
- nothing pending. Only the latest run per check name counts (re-runs report several). A later run
of the same name in another check suite never replaces an earlier one (two workflows can have
jobs of the same name, and a failure of one is never hidden by the other); a re-run of failed
jobs stays in its run's suite and does;
- a required check counts only as a check run of the `github-actions` app;
- a check run whose name starts with `omni-queue/` is not part of the verdict;
- commit statuses are not part of the verdict at all: the tick does not read them for it, whoever
@@ -539,6 +553,17 @@ queue deliberately reads none (see below):
so without this rule that PR's author could cancel, fail or re-run checks on it at will. Check
runs that come from no Actions workflow run (other apps) are not read either. Until GitHub
lists a push run for the ref, the verdict stays pending;
- inside those suites, only the check runs of the push runs' **jobs** count. The tick lists the
jobs of each push run of the ref (`listWorkflowRunJobs`:
`GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs?filter=all`, every attempt, so the jobs a
re-run adds still count) and keeps the check runs those jobs name (each job's `check_run_url`).
Any token with `checks: write` (the `GITHUB_TOKEN` of any workflow run in the repository) can
create a check run on the candidate SHA, and GitHub files it in one of the commit's Actions
suites. Such a run is no job: it is ignored for the verdict (neither green nor red), and the
first tick that meets it for a batch records one `error` event,
``check runs on the candidate that are not jobs of its push runs, ignored for the verdict: `<name>` (app `<slug>`), …``.
A job listing that fails fails the tick: nothing is blessed. The listing costs one request per
push run of the ref and tick (two today: `ci.yml` and `quality.yml`), one more per 100 jobs;
- no GitHub Actions run at all `NO_RUN_TIMEOUT_MINUTES` (45) after the push → `timeout` with
reason `no-ci-run` (the push triggered nothing), handled like an infra failure;
- not decided after `checksTimeoutMinutes` (240) → `timeout`, handled like an infra failure.
@@ -550,7 +575,9 @@ A required check concluded `skipped` or `neutral` is red, not green.
A candidate runs the same gates as the nightly Release-Green of the release branch; what follows is
where the two differ. For the external-state gates, the `check:test-masking` step and the
candidate's AI-attribution range, a candidate is a `push` to `omni-queue/**` and only that event: a
`workflow_dispatch` on such a ref runs them as any other run does, every one of them blocking.
`workflow_dispatch` on such a ref runs the external-state gates blocking, as any other run does, and
skips the two candidate-only steps (`Detect test-masking against the omni-queue candidate base` and
`AI attribution in the omni-queue candidate range (Hard Rule #16)`).
Older conditions of `ci.yml` and `quality.yml` test the ref alone (the candidate base, the
base-relative ratchets, the advisory i18n drift path), so a dispatch on such a ref takes those.
@@ -616,7 +643,9 @@ blaming](#re-running-the-failed-jobs-before-blaming-flakyreruns)):
out; a gate that fails alone still blocks. The same SHA is retried under a new ref up
to `infraRetries` times (once in the shipped config); the infra failure that comes after the last
retry (with `infraRetries: 0`, the first one) pauses the base with reason `infra`, except when
every run of the candidate only hit a job time limit (see 5).
every run of the candidate only hit a job time limit (see 5), and except for a base repair under
a `base-red` pause: that batch is dropped and the `base-red` pause stays (see [The two
lanes](#the-two-lanes-lanes)).
2. **Needs a base check** — the batch is red on a base SHA that has no finished verdict (green, red
or timeout). The tick pushes a `basecheck-<sha12>` candidate with the bare base SHA (same CI
profile), records `base-check-started` and parks the red batch in `inFlight.pendingRed`. This
@@ -635,8 +664,10 @@ blaming](#re-running-the-failed-jobs-before-blaming-flakyreruns)):
check (shards re-balance, so failures grouped differently still match). The batch is not
guilty: the base is paused with reason `base-red` and the pause lifts by itself when the base
tip moves (someone landed the fix), or, in live mode, when a PR of the base-repair lane lands
(`resumed`, detail `base repaired by #<n>`); neither lift happens while
`landWithdrawalsOverflow` holds entries (see [CLI](#cli)). While it lasts the queue processes
(`resumed`, detail `base repaired by #<n>`). A `base-red` pause never stands together with
entries in `landWithdrawalsOverflow`: a tick that finds both (a hand-edited state) replaces it
with the full-list pause first (see [Paused: landWithdrawals is
full](#paused-landwithdrawals-is-full)). While it lasts the queue processes
only that lane (see [Batch selection](#batch-selection), which also says what happens to a
base-repair candidate that fails like the base). When a
`🔴 Release branch not green: <base>` issue is open, the queue comments on it.
@@ -743,31 +774,29 @@ permission the job asks for has not been measured: it may fail every tick at the
may bypass those rules: the App's token has no Administration permission, and anyone who can
edit a ruleset's bypass list can disable the ruleset as well. A rule that comes from another
ruleset with the same types counts as protection.
- **A landing that always fails on a base that keeps moving is never bounded.** Each time the base
- **A landing that always fails on a base that keeps moving is never paused.** Each time the base
moves, the batch is recomposed: a new candidate with no landing count. With a landing that fails
without a verdict every time, each new candidate is blessed once, its tick fails and withdraws
it, and the next tick finds the base moved and recomposes. No pause and no stalled alert: the
failed ticks alternate with ticks that complete, so the failed-tick count never reaches three,
and every new candidate starts the in-flight clock again. The `land-blessed` event of these
it, and the next tick finds the base moved and recomposes. No pause: the attempts limit counts
per candidate. But every blessing also counts in `landingsWithoutVerdict`, the base's run of
landings without a verdict (ended by a landing that answers, a `landed` batch, or an operator's
pause or resume), and at 3 the queue is stalled (see [Health](#health)): one alert mentions the
approvers. The `land-blessed` event of these
blessings is never stored: the tick records it after its mid-tick save, then fails, and the
failure path of `main()` saves only its own `error` event (and the failed-tick count) onto the
stored document, which holds what the mid-tick save wrote. What the stored history shows,
repeated, is `checks-green`, then `error` (the error the landing threw, such as
`land: commit <sha> is not available from origin`), then `land-rejected`
(`base moved: …; recomposing`), then `batch-started`.
- **Stored withdrawals alone do not bring a tick to a base.** A tick run without a base visits a
base for its labeled PRs, a batch in flight, a pause, a `stalled` episode or a `failedTicks` count
(`discoverBases` in `scripts/merge-queue/tick.mjs`); stored withdrawals are not on that list, and
the watchdog uses the same notion of work. So on a base whose only work is a stored withdrawal (a
batch that was dropped while GitHub refused its withdrawal, no other labeled PR, no pause), no
tick runs: the entry is not retried, the App's `success` stays on that commit, the dashboard stops
changing and nobody is told. What to look at: the dashboard's line
`Stored land-status withdrawals (retried on every tick): …`, the line `stored withdrawals: …` of
`npm run queue -- status`, or the fields `landWithdrawals` and `landWithdrawalsOverflow` of
`state/<sanitized-base>.json`. What to do: pause the base (`npm run queue -- pause <base>`: a
paused base is visited, and its ticks retry both lists), or dispatch a tick for that base from
the Actions tab (workflow `omni-queue`, inputs `base` and `action` `tick`). This is a defect, to
be fixed in the code.
- **A base without labeled PRs is found only through its open dashboard issue.** A tick run without
a base visits a base for its labeled PRs or, through its open dashboard issue, for a batch in
flight, a pause, a stored withdrawal, a `stalled` episode or a `failedTicks` count (`discoverBases`
in `scripts/merge-queue/tick.mjs`). Do not close a dashboard issue. Every tick that completes
creates a new one when none is open, and so does a tick that fails while its state stores
withdrawals or has an alert to post; but if a person closes it after the last tick, a base with
work and no labeled PR is not visited until a tick is dispatched for it (workflow `omni-queue`,
inputs `base` and `action` `tick`). Its state then stops moving, and after 90 minutes the watchdog
reports the base (see [Who is told, and when](#who-is-told-and-when)).
- **"Already landed" is tip equality, and only with `merge-ff`.** A batch whose landing went
through while its state save was lost is recognized only when its candidate is still the base
tip. If the base moved again after that (a merge by hand on top), the batch is taken as not
@@ -775,17 +804,6 @@ permission the job asks for has not been measured: it may fail every tick at the
`landed` event is recorded and the counters are short by one; its PRs are merged, so they are not
batched again. With `squash-api` such a batch is never recognized: a pause or a later tick writes
`failure` on its PR heads (merged already) and records no `landed` event.
- **With `squash-api`, a rate limit on a merge call pauses the base.** Unlike the limits above,
this one is a defect, to be fixed before `squash-api` is used. The shipped mode is
`merge-ff`, where a rate-limited landing fails the tick and a later tick lands again (within
`LANDING_ATTEMPTS_LIMIT`, see [Who lands](#who-lands-landingactor)). With
`squash-api` the rate-limit mark of the error is lost inside `landSquashApi`: once the GitHub
client's own retries run out, the call is a failed merge (`merge-failed`), and the base is paused
with reason `error`. With `actor: "owner"`, when GitHub's message lacks the words `rate limit`
(a 403 with `Retry-After`, or with `x-ratelimit-remaining: 0`), the detail also carries the
land-token hint `(check OMNI_QUEUE_LAND_TOKEN: landing.actor is "owner")`, although the token is
fine. The PRs merged before the refused one are on the base and named on the `error` event. Wait
for the limit to reset, then `npm run queue -- resume <base>`.
- **The tree recomposition depends on the runner's git.** The composition check recomposes each
merge commit of the candidate with `git merge-tree` on the tick's runner. A different git version
on the runner that composed the candidate and on the runner that checks it could, in principle,
@@ -815,17 +833,18 @@ candidate in flight (refs, SHAs, PRs, stage, attempts, bisection), the last 10 b
shadow bookkeeping, counters and the last 200 history events. Optional fields, each absent until
it is needed:
| Field | What it holds | Explained in |
| --------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------- |
| `inFlight.landAttempts` | landings of the candidate that ended without a verdict | below, and [Who lands](#who-lands-landingactor) |
| `inFlight.rerun` | the re-run of the candidate's failed jobs | [Re-running](#re-running-the-failed-jobs-before-blaming-flakyreruns) |
| `inFlight.lane` | the lane of a lane batch (`hotfix` or `base-repair`) | [Batch selection](#batch-selection) |
| `inFlight.freezeHold` | a green batch held by a release freeze (`since`, `endedAt`) | [Health](#health) |
| `landWithdrawals` | refused `failure` land statuses, retried | below |
| `landWithdrawalsOverflow` | the same, for refusals that found `landWithdrawals` full | below |
| `batchWindow` | the adaptive batch size | [Batch selection](#batch-selection) |
| `repairTried` | base-repair heads that failed like the base (PR number → head SHA) | [Batch selection](#batch-selection) |
| `failedTicks`, `failedSince`, `stalled` | the count of failed ticks in a row, and the open stalled episode | [Health](#health) |
| Field | What it holds | Explained in |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `inFlight.landAttempts` | landings of the candidate that ended without a verdict | below, and [Who lands](#who-lands-landingactor) |
| `inFlight.rerun` | the re-run of the candidate's failed jobs | [Re-running](#re-running-the-failed-jobs-before-blaming-flakyreruns) |
| `inFlight.lane` | the lane of a lane batch (`hotfix` or `base-repair`) | [Batch selection](#batch-selection) |
| `inFlight.freezeHold` | a green batch held by a release freeze (`since`, `endedAt`) | [Health](#health) |
| `landWithdrawals` | refused `failure` land statuses, retried | below |
| `landWithdrawalsOverflow` | the same, for refusals that found `landWithdrawals` full | below |
| `batchWindow` | the adaptive batch size | [Batch selection](#batch-selection) |
| `repairTried` | base-repair heads that failed like the base (PR number → head SHA) | [Batch selection](#batch-selection) |
| `failedTicks`, `failedSince`, `stalled` | the count of failed ticks in a row, and the open stalled episode (with `stalled.quietTicks` while an episode of failed ticks is held) | [Health](#health) |
| `landingsWithoutVerdict` | the base's run of landings that ended without a verdict (`count`, `since`) | [Health](#health) |
Three fields belong to the land status (see [Who lands](#who-lands-landingactor)):
@@ -867,14 +886,17 @@ text (a pause detail that names commits, an error that names several PRs).
it, and repeating a tick is safe.
- **An unchanged state is not written again**: most ticks change nothing (idle queue, candidate
checks still pending), and a commit per tick would add hundreds of commits a day to the state
branch. A tick saves only when the document changed. When it did not, the tick writes a
branch. A tick saves only when the document changed (the `attempts` count of a stored withdrawal
alone is no change). When it did not, the tick writes a
heartbeat at most once per hour (`STATE_HEARTBEAT_MINUTES` in `scripts/merge-queue/tick.mjs`),
which only moves `updatedAt`. A tick run without a base (every tick but a dispatch that names
one) visits a managed base that has open PRs with the queue label (drafts included), and a base
whose open dashboard issue leads to a state with a candidate in flight, a pause, a `stalled`
episode or a `failedTicks` count (one completed tick clears the last two; then a base without
work drops out). Stored land-status withdrawals do not make a base visited (see the known limits
in [Red candidates](#red-candidates-classification-base-check-bisection)). A dispatch with a base
whose **open** dashboard issue leads to a state with a candidate in flight, a pause, a stored
land-status withdrawal, a `stalled` episode or a `failedTicks` count (a completed tick clears the
count, and the episode once nothing is stalled: at once, or after two completed ticks in a row
for an episode that failed ticks opened; then a base without work drops out). Do not close a
dashboard issue (see the known limits in
[Red candidates](#red-candidates-classification-base-check-bisection)). A dispatch with a base
visits that base whatever it holds. For a visited base, an `updatedAt` older than an hour plus
one tick interval means no tick is completing (see [Health](#health)).
- **Invalid documents are never reset silently**: the tick fails loudly, and so does the failure
@@ -896,13 +918,12 @@ the body changed (the `Last change` footer line aside). From top to bottom:
(see [Health](#health));
- the pause banner, when the base is paused: reason, since, the detail (cut at 300 characters), the
base tip when paused, the issue, and what to do. For `base-red` it reads "Resumes automatically
when the base tip moves and turns green": the pause lifts when the tip moves (see
when the base tip moves.": the pause lifts when the tip moves (see
[Red candidates](#red-candidates-classification-base-check-bisection), item 3); whether the new
tip is green is decided by the next red batch, if any. The `base-red` banner says this even in
the one case where the pause does not lift by itself: a `base-red` pause while
`landWithdrawalsOverflow` holds entries (reachable only from a hand-edited state; see
[CLI](#cli)). While `landWithdrawalsOverflow` holds entries, any other banner says a resume is
refused and how many entries remain;
tip is green is decided by the next red batch, if any. A `base-red` pause never stands together
with entries in `landWithdrawalsOverflow` (a tick replaces it with the full-list pause). While
`landWithdrawalsOverflow` holds entries, the banner says a resume is refused and how many
entries remain;
- the stored land-status withdrawals, when either list holds entries:
`Stored land-status withdrawals (retried on every tick): <n> in landWithdrawals, <m> in landWithdrawalsOverflow.`;
- **In flight**: the batch or base check, its lane, stage, start and infra retries, the candidate
@@ -912,8 +933,9 @@ the body changed (the `Last change` footer line aside). From top to bottom:
- **Metrics**: PRs landed, CI runs per merged PR, red batches, bisections, and the median and p90
time from approval to landing (to `would-land` in shadow mode).
A tick that fails creates the dashboard issue only when it is missing, to post a stalled alert;
that body says the queue was not read and its heartbeat line is
A tick that fails creates the dashboard issue only when none is open and it has a stalled alert to
post or its state stores land-status withdrawals (so the next scheduled tick finds the base); that
body says the queue was not read and its heartbeat line is
`Heartbeat: none: a tick that failed wrote this dashboard`. A dashboard that stopped changing is
an incident (see [Dashboard stopped updating](#dashboard-stopped-updating)).
@@ -928,12 +950,14 @@ dashboard that moves; an hour that stopped moving means no tick of that base has
**Stalled.** The queue is stalled when any of these holds (each limit exclusive):
| Rule | Holds when | Constant |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| paused | the base has been paused for more than 120 minutes, whatever the reason (a manual pause too: it may be forgotten) | `STALL_PAUSED_MINUTES` |
| in flight | a candidate has been in flight for more than `checksTimeoutMinutes` + 60 (300 in the shipped config), counted from its start | `STALL_INFLIGHT_GRACE_MINUTES` |
| waiting | eligible PRs, nothing in flight and no pause, for more than `batch.maxWaitMinutes` + 30, counted from the oldest eligible approval or the last event that is not `error` or `queue-stalled`, whichever is later | `STALL_WAITING_GRACE_MINUTES` |
| ticks fail | 3 ticks in a row failed (`failedTicks`) | `STALL_ERROR_STREAK` |
| Rule | Holds when | Constant |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| paused | the base has been paused for more than 120 minutes, whatever the reason (a manual pause too: it may be forgotten); a pause with reason `error` or `tree-divergence` at once, from the first tick, since nothing but an operator lifts it | `STALL_PAUSED_MINUTES`, `STALL_AT_ONCE_PAUSE_REASONS` |
| in flight | a candidate has been in flight for more than `checksTimeoutMinutes` + 60 (300 in the shipped config), counted from its start | `STALL_INFLIGHT_GRACE_MINUTES` |
| waiting | eligible PRs, nothing in flight and no pause, for more than `batch.maxWaitMinutes` + 30, counted from the oldest eligible approval or the last event that is not `error` or `queue-stalled`, whichever is later | `STALL_WAITING_GRACE_MINUTES` |
| ticks fail | 3 ticks in a row failed (`failedTicks`) | `STALL_ERROR_STREAK` |
| withdrawal | an entry of `landWithdrawals` or `landWithdrawalsOverflow` was first refused more than 60 minutes ago (since: the oldest such entry's `since`) | `STALL_WITHDRAWAL_MINUTES` |
| landings | 3 landings in a row ended without a verdict and none landed since (`landingsWithoutVerdict`, counted before each blessing whatever the candidate) | `STALL_LANDINGS_WITHOUT_VERDICT` |
A green batch held by a release freeze is waiting, not stalled: `inFlight.freezeHold` records the
hold (`since`, set by the first held tick; `endedAt`, set by the first tick that does not hold it
@@ -952,8 +976,20 @@ refused withdrawal) do not count.
comment on the dashboard issue that mentions the approvers. Until that comment is posted
(`notified` in `stalled`), every tick, completed or failed, tries again; a failed tick creates the
dashboard issue when there is none. While the episode lasts nothing more is raised, even if its
reasons change. A completed tick whose verdict is not stalled ends it (the field goes); a failed
tick never ends one. A base whose queue stalls again later gets a new episode and a new comment.
reasons change. A completed tick whose verdict is not stalled ends it (the field goes), except an
episode that failed ticks opened (its reasons name `ticks in a row failed`): that one ends only after
two completed ticks in a row found no stall (`STALL_CLEAR_COMPLETED_TICKS` in
`scripts/merge-queue/tick.mjs`; `quietTicks` in `stalled` counts them, and a failed tick or a
stalled verdict in between starts the count again), so failures broken by one completed tick are
one episode with one comment. A failed tick never ends one. A base whose queue stalls again later
gets a new episode and a new comment.
**`landingsWithoutVerdict`.** The tick counts each landing attempt in the save before its blessing
(`count`, and `since` at the first) and deletes the field when a landing answers (any verdict: landed,
refused, the base moved), when a batch lands (also one the next tick finds landed), and on an
operator's pause or resume. A landing that throws or a tick that dies after the blessing leaves it
counted, whatever the candidate, so a base that keeps moving under a landing that keeps failing is
reported although each new candidate starts its own `landAttempts`.
Who else is told, and when: the watchdog workflow and its issue, see [Who is told, and
when](#who-is-told-and-when). What to do for each reason: see [The queue
@@ -967,33 +1003,33 @@ history holds a type the code does not know is refused, which matters for a roll
[Cut-over and rollback](#cut-over-and-rollback)). `tests/unit/merge-queue/runbook.test.mjs` fails
when this table and `EVENT_TYPES` differ.
| Event | When it happens | What the operator does |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batch-started` | A candidate was composed and pushed: a new batch, a recomposition or a bisection segment. The detail names the PRs and the base, and for a root batch with `batch.adaptive` the window it was cut to (`window W`). A lane batch names its lane instead of a window (`lane hotfix` or `lane base-repair`). | Nothing. Its CI is linked from the dashboard. |
| `pr-skipped-conflict` | A PR conflicts with an earlier PR of the same batch. It is left out of this candidate and stays queued for a later batch. | Nothing. If it repeats, the author rebases the PR over the one it conflicts with. |
| `pr-dequeued` | A PR left the queue: `base-conflict` (it no longer merges into the base: label removed, comment), `already-in-base` (both labels removed, comment), or `withdrawn: <why>` before composition (label, head or approval changed). In shadow mode the first two are `would-dequeue` events instead. | For `base-conflict` the author updates the branch and an approver labels it again; the other cases need nothing. |
| `checks-green` | Every required check of the candidate (or of a base check) concluded `success`. | Nothing. The landing (or `would-land`) follows. |
| `checks-red` | A check of the candidate failed. The detail names the failing checks; on a base check it starts with `base-check red`. It ends with `; re-run refused (<status or reason>)` when the re-run of the failed jobs was refused. A batch red again after its re-run records a second one. | Nothing yet: classification follows (a re-run of the failed jobs, infra retry, base check, bisection or culprit). |
| `flaky-rerun` | A red batch or bisection segment that would be bisected, tested against the base or dequeued: its failed jobs are re-run once after its runs end (`flakyReruns`). Detail: `re-running the failed jobs of <batch> once (<jobs>) before <bisecting\|a base check\|dequeuing #n>` (five jobs at most). A green re-run's `checks-green` reads `green after re-running <jobs> (flaky)`. | Nothing. A real failure is blamed later: after the rest of the CI run and the re-run. Jobs that often turn green on the re-run are flaky tests to fix. |
| `checks-timeout` | No verdict after `checksTimeoutMinutes`, or no CI run at all after 45 minutes (`no CI run for candidate`). | For `no CI run`, check the Actions policies and the `omni-queue/**` triggers of `ci.yml` / `quality.yml` (see [Infra failure or timeout](#infra-failure-or-timeout)). |
| `land-blessed` | Live mode: the App wrote the land status (`landing.statusContext`, `success`) on the commits about to land, right before the landing. | Nothing. A `landed` event should follow; if a pause follows, the status was withdrawn. |
| `manual-bless` | An approver's `bless`: the App wrote the land status on one chosen commit. The detail names who and why. | Push or merge that commit yourself (see [Land status and manual bless](#land-status-and-manual-bless)). |
| `landed` | Live mode: the batch reached the base (the `merge-ff` push or every `squash-api` merge). | Nothing. |
| `would-land` | Shadow mode: a green batch that live mode would have landed. The detail can add `merged meanwhile by another path`, `live mode would pause: queue refs unprotected (…)` and, after the name of a check, `could not run (…): live mode would fail the tick`. | Compare with what Mergify merged. Both notes must be gone before `mode: "live"`: apply the queue-refs ruleset, or fix the check that could not run. |
| `would-dequeue` | Shadow mode: live mode would have removed this PR from the queue (label kept, no comment). The detail gives the reason code(s). | Nothing during shadow. |
| `land-rejected` | The batch was not landed: the base moved (`base moved…; recomposing`: the same PRs are recomposed on the tip), the batch changed before landing (`batch changed before landing: …`: the PRs left are recomposed), or a release-freeze holds it (`base-frozen: …; <batch> waits for the freeze to lift`, once per batch: it stays in flight, not recomposed, and lands after the freeze). | Nothing for an occasional one; frequent ones mean something else merges into the base (see [Recomposition when the base moves](#recomposition-when-the-base-moves)). |
| `bisect-split` | A red batch (or a red segment of a bisection) is split; the detail names the segment tested next. | Nothing. |
| `culprit-found` | Bisection or a batch of one isolated the PR that fails. Live: label removed, comment. Shadow, or a head that moved: label kept. | The author fixes the PR and an approver labels it again. |
| `base-check-started` | A red batch on a base with no verdict: the bare base is pushed as a `basecheck-<sha12>` candidate. | Nothing. |
| `base-red-paused` | The base check is red with the same failures as the batch: the base is paused with reason `base-red`. | Fix the base (see [Base-red](#base-red)). The pause lifts when the tip moves. |
| `hang-suspected` | A job hit its time limit on every run of the candidate: treated as a hanging test (base check, then bisection) instead of an infra pause. | Nothing yet; look at the culprit comment when one is named. |
| `infra-retry` | Every failure is infrastructure (or the checks timed out): the same SHA is retried under a new ref (`-r1`, `-r2`, …), while retries of `infraRetries` are left. The detail reads `retry <n>/<infraRetries>: …`. | Nothing. |
| `infra-paused` | An infrastructure failure (or a timeout) with no retry of `infraRetries` left: the base is paused with reason `infra`. With the shipped `infraRetries: 1` that is the second failure; with `0`, the first. When every run only hit a job time limit, the outcome is `hang-suspected` instead (see its row). | Check the Actions status and the candidate's run, then `npm run queue -- resume <base>`. |
| `paused` | A manual pause (`npm run queue -- pause`, or `action=pause`). | `npm run queue -- resume <base>` when ready. |
| `resumed` | A manual resume (`manual resume (was <reason>)`, followed by `; the pause it cleared named commits kept nowhere (both lists full), check their <landing.statusContext> by hand: <sha> (<batchId>), …` when that pause named such commits), a `base-red` pause lifted because the base tip moved (`base moved <sha12> → <sha12> after a base-red pause`), or, in live mode, a `base-red` pause lifted because a PR of the base-repair lane landed (`base repaired by #<n>`). Neither automatic lift happens while `landWithdrawalsOverflow` holds entries. | For a detail that names commits kept nowhere: check their land status by hand (see [Paused: landWithdrawals is full](#paused-landwithdrawals-is-full)). Otherwise nothing. |
| `tree-divergence` | `squash-api` only: the base tree after the merges differs from the candidate tree, or an open divergence issue stops a landing. The base is paused with reason `tree-divergence`. | Compare the trees, fix the base, close the issue, resume (see [Tree divergence](#tree-divergence-squash-api-only)). |
| `error` | A tick failed (the detail is its error message); a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused (`<batchId>: could not withdraw …`); a stored withdrawal whose attempts reached 288 (`… has failed 288 times since …; the queue keeps retrying it`); a refused withdrawal that found the `landWithdrawals` list (20) full, kept in `landWithdrawalsOverflow` (`landWithdrawals is full …`; it pauses a base that is not paused, or replaces a `base-red` pause); a refused resume (`resume refused: …`, the pause stays); a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed. The detail says which. | Read the detail: each pause detail, and each other `error` detail an operator must act on, has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
| `queue-stalled` | The queue stopped making progress: paused for more than 120 minutes, a candidate in flight for more than `checksTimeoutMinutes` + 60, eligible PRs and nothing started for more than `batch.maxWaitMinutes` + 30, or three ticks in a row that failed. One per episode, with one comment on the dashboard issue that mentions the approvers (see [Health](#health)). The detail lists the reasons. | See [The queue stalled](#the-queue-stalled). |
| Event | When it happens | What the operator does |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `batch-started` | A candidate was composed and pushed: a new batch, a recomposition or a bisection segment. The detail names the PRs and the base, and for a root batch with `batch.adaptive` the window it was cut to (`window W`). A lane batch names its lane instead of a window (`lane hotfix` or `lane base-repair`). | Nothing. Its CI is linked from the dashboard. |
| `pr-skipped-conflict` | A PR conflicts with an earlier PR of the same batch. It is left out of this candidate and stays queued for a later batch. | Nothing. If it repeats, the author rebases the PR over the one it conflicts with. |
| `pr-dequeued` | A PR left the queue: `base-conflict` (it no longer merges into the base: label removed, comment), `already-in-base` (both labels removed, comment), or `withdrawn: <why>` before composition (label, head or approval changed). In shadow mode the first two are `would-dequeue` events instead. | For `base-conflict` the author updates the branch and an approver labels it again; the other cases need nothing. |
| `checks-green` | Every required check of the candidate (or of a base check) concluded `success`. | Nothing. The landing (or `would-land`) follows. |
| `checks-red` | A check of the candidate failed. The detail names the failing checks; on a base check it starts with `base-check red`. It ends with `; re-run refused (<status or reason>)` when the re-run of the failed jobs was refused. A batch red again after its re-run records a second one. | Nothing yet: classification follows (a re-run of the failed jobs, infra retry, base check, bisection or culprit). |
| `flaky-rerun` | A red batch or bisection segment that would be bisected, tested against the base or dequeued: its failed jobs are re-run once after its runs end (`flakyReruns`). Detail: `re-running the failed jobs of <batch> once (<jobs>) before <bisecting\|a base check\|dequeuing #n>` (five jobs at most). A green re-run's `checks-green` reads `green after re-running <jobs> (flaky)`. | Nothing. A real failure is blamed later: after the rest of the CI run and the re-run. Jobs that often turn green on the re-run are flaky tests to fix. |
| `checks-timeout` | No verdict after `checksTimeoutMinutes`, or no CI run at all after 45 minutes (`no CI run for candidate`). | For `no CI run`, check the Actions policies and the `omni-queue/**` triggers of `ci.yml` / `quality.yml` (see [Infra failure or timeout](#infra-failure-or-timeout)). |
| `land-blessed` | Live mode: the App wrote the land status (`landing.statusContext`, `success`) on the commits about to land, right before the landing. | Nothing. A `landed` event should follow; if a pause follows, the status was withdrawn. |
| `manual-bless` | An approver's `bless`: the App wrote the land status on one chosen commit. The detail names who and why. | Push or merge that commit yourself (see [Land status and manual bless](#land-status-and-manual-bless)). |
| `landed` | Live mode: the batch reached the base (the `merge-ff` push or every `squash-api` merge). | Nothing. |
| `would-land` | Shadow mode: a green batch that live mode would have landed. The detail can add `merged meanwhile by another path`, `live mode would pause: queue refs unprotected (…)` and, after the name of a check, `could not run (…): live mode would fail the tick`. | Compare with what Mergify merged. Both notes must be gone before `mode: "live"`: apply the queue-refs ruleset, or fix the check that could not run. |
| `would-dequeue` | Shadow mode: live mode would have removed this PR from the queue (label kept, no comment). The detail gives the reason code(s). | Nothing during shadow. |
| `land-rejected` | The batch was not landed: the base moved (`base moved…; recomposing`: the same PRs are recomposed on the tip), the batch changed before landing (`batch changed before landing: …`: the PRs left are recomposed), or a release-freeze holds it (`base-frozen: …; <batch> waits for the freeze to lift`, once per batch: it stays in flight, not recomposed, and lands after the freeze). | Nothing for an occasional one; frequent ones mean something else merges into the base (see [Recomposition when the base moves](#recomposition-when-the-base-moves)). |
| `bisect-split` | A red batch (or a red segment of a bisection) is split; the detail names the segment tested next. | Nothing. |
| `culprit-found` | Bisection or a batch of one isolated the PR that fails. Live: label removed, comment. Shadow, or a head that moved: label kept. | The author fixes the PR and an approver labels it again. |
| `base-check-started` | A red batch on a base with no verdict: the bare base is pushed as a `basecheck-<sha12>` candidate. | Nothing. |
| `base-red-paused` | The base check is red with the same failures as the batch: the base is paused with reason `base-red`. | Fix the base (see [Base-red](#base-red)). The pause lifts when the tip moves. |
| `hang-suspected` | A job hit its time limit on every run of the candidate: treated as a hanging test (base check, then bisection) instead of an infra pause. | Nothing yet; look at the culprit comment when one is named. |
| `infra-retry` | Every failure is infrastructure (or the checks timed out): the same SHA is retried under a new ref (`-r1`, `-r2`, …), while retries of `infraRetries` are left. The detail reads `retry <n>/<infraRetries>: …`. | Nothing. |
| `infra-paused` | An infrastructure failure (or a timeout) with no retry of `infraRetries` left: the base is paused with reason `infra`. With the shipped `infraRetries: 1` that is the second failure; with `0`, the first. When every run only hit a job time limit, the outcome is `hang-suspected` instead (see its row). A base repair under a `base-red` pause is dropped instead, with an `error` event, and the `base-red` pause stays. | Check the Actions status and the candidate's run, then `npm run queue -- resume <base>`. |
| `paused` | A manual pause (`npm run queue -- pause`, or `action=pause`). The detail is the reason, on one line, cut at 200 characters. | `npm run queue -- resume <base>` when ready. |
| `resumed` | A manual resume (`manual resume (was <reason>)`, followed by `; the pause it cleared named commits kept nowhere (both lists full), check their <landing.statusContext> by hand: <sha> (<batchId>), …` when that pause named such commits), a `base-red` pause lifted because the base tip moved (`base moved <sha12> → <sha12> after a base-red pause`), or, in live mode, a `base-red` pause lifted because a PR of the base-repair lane landed (`base repaired by #<n>`). Neither automatic lift happens while `landWithdrawalsOverflow` holds entries: a tick replaces such a `base-red` pause with the full-list pause. | For a detail that names commits kept nowhere: check their land status by hand (see [Paused: landWithdrawals is full](#paused-landwithdrawals-is-full)). Otherwise nothing. |
| `tree-divergence` | `squash-api` only: the base tree after the merges differs from the candidate tree, or an open divergence issue stops a landing. The base is paused with reason `tree-divergence`. | Compare the trees, fix the base, close the issue, resume (see [Tree divergence](#tree-divergence-squash-api-only)). |
| `error` | A tick failed (the detail is its error message); a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused (`<batchId>: could not withdraw …`); a stored withdrawal refused for a day (`… has been refused for more than 24 h, since …; the queue keeps retrying it`); a refused withdrawal that found the `landWithdrawals` list (20) full, kept in `landWithdrawalsOverflow` (`landWithdrawals is full …`; it pauses a base that is not paused, or replaces a `base-red` pause, also one a tick loads together with overflow entries); a refused resume (`resume refused: …`, the pause stays); a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed; check runs on the candidate that are no job of its push runs (`check runs on the candidate that are not jobs of its push runs, ignored for the verdict: …`, once per batch); a base repair dropped after an infrastructure failure under a `base-red` pause (`… failed for infrastructure after … dropped; the base-red pause stays`). A tick that hit a rate limit while landing fails with `landing <batchId> hit a rate limit (…)`. The detail says which. | Read the detail: each pause detail, and each other `error` detail an operator must act on, has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
| `queue-stalled` | The queue stopped making progress: paused for more than 120 minutes (with reason `error` or `tree-divergence`, at once), a candidate in flight for more than `checksTimeoutMinutes` + 60, eligible PRs and nothing started for more than `batch.maxWaitMinutes` + 30, three ticks in a row that failed, a stored withdrawal refused for more than 60 minutes, or three landings in a row without a verdict. One per episode, with one comment on the dashboard issue that mentions the approvers (see [Health](#health)). The detail lists the reasons. | See [The queue stalled](#the-queue-stalled). |
## Configuration keys
@@ -1062,16 +1098,21 @@ What the read commands show:
withdrawal lists when either holds entries, the batch in flight with its stage, the counters,
the age of the last state save, and the labeled PRs (drafts marked `[draft]`). The save line
adds `STALE: no tick has saved it for more than 90 min while the base has work` when the base
has a candidate in flight, a pause or a labeled PR and its state is older than 90 minutes
has a candidate in flight, a pause, a stored withdrawal or a labeled PR and its state is older
than 90 minutes
(`STATUS_STALE_MINUTES`).
- `history` prints the last 20 events (`--limit` from 1 to 9999), newest first, each detail cut at
200 characters.
- `why` evaluates the eligibility rules for one PR, with the local config, and prints the reasons,
the approval and the actions the next tick would take. It does not read the PR gates
(`prChecksRequired`): a PR the dashboard shows as `pr-checks-not-green` can read `ELIGIBLE`
here.
- `why` evaluates the eligibility rules for one PR, with the local config, as a tick does: the PR
gates (`prChecksRequired`, read from the head's check runs as the tick reads them) and the lane (a
base repair's PR gates do not apply). It prints the verdict, the reasons, the approval,
`PR gates (prChecksRequired): <gate> <state>; …`, `lane: <hotfix|base-repair|none>` and the
actions the next tick would take.
`pause` without `--reason` records the detail `paused from the CLI`.
`pause` without `--reason` records the detail `paused from the CLI`. The tick stores a reason on one
line, cut at 200 characters (`OPERATOR_REASON_CHARS` in `scripts/merge-queue/tick.mjs`), with the
words `Kept nowhere` written `Kept-nowhere`, so that no reason can be read as the queue's own
`Kept nowhere (both lists full)` clause.
`pause`, `resume` and `bless` dispatch the workflow and then wait until the base's state document
shows the action applied (a new `paused` event, no pause left, or a new `manual-bless` event for
@@ -1104,12 +1145,11 @@ from the Actions tab that the tick refuses ends as a green run: look for the dec
(`npm run queue -- history --base <base>`), and the dashboard's pause banner. There is no flag to
force a resume: an entry GitHub refuses for good is handled by the last resort of [Paused:
landWithdrawals is full](#paused-landwithdrawals-is-full). The dashboard's pause banner says to
wait, with the number of overflow entries, instead of how to resume. The automatic lifts of a
`base-red` pause (the tip moved, a repair landed) follow the same rule: the queue never writes a
`base-red` pause together with overflow entries, but a hand-edited state can hold both. Then the
pause stays while the overflow holds entries. The tick checks a moved tip before its retry pass, so
the lift comes on the tick after the one that empties the overflow; a repair that landed meanwhile
moved the tip, so that lift is recorded as `base moved …`, not `base repaired by #<n>`.
wait, with the number of overflow entries, instead of how to resume. A `base-red` pause, which lifts
by itself, never stands together with overflow entries: the queue never writes both, and a tick that
loads both (a hand-edited state) replaces the `base-red` pause with the full-list pause before
anything could lift it or bless its repair lane; that pause lifts only through a resume, under the
rule above.
The CLI evaluates eligibility with the local config file (`config/merge-queue/config.json`, or the
file in OMNI_QUEUE_CONFIG, an override for the sandbox and the tests that the workflow never sets),
@@ -1120,6 +1160,12 @@ The same `pause`, `resume`, `dry-run` and `bless` actions are available from the
workflow `omni-queue` → "Run workflow" → inputs `base`, `action`, `reason` and, for `bless` only,
`sha`. `dry-run` observes and decides, prints the plan and writes nothing.
Who may do what: a `pause` or a `resume` needs only the right to dispatch the workflow (write access
to the repository); the tick checks no approver for them. Only `bless` checks that the dispatcher
and the triggering actor are both in `approvers`. A resume never blesses anything by itself: the
batch in flight is followed again, and before it is blessed every check of [Before the land status
is written](#before-the-land-status-is-written) runs again.
## Shadow mode and live mode
- **`shadow`** (default): the queue composes candidates, runs the full CI on them, bisects and
@@ -1219,9 +1265,8 @@ queue to shadow (or shadow to live), turning the kill switch off or rolling back
`stage <stage>` for the batch in flight) and that both `landWithdrawals` and
`landWithdrawalsOverflow` are empty: the fields are absent from `state/<sanitized-base>.json` on
`omni-queue-state` (an empty list leaves no field). While entries remain, wait for the next ticks
to retry them and check again: the pause of step 1 makes sure ticks visit the base (a base whose
only work is stored withdrawals is not visited, see the known limits in [Red
candidates](#red-candidates-classification-base-check-bisection)). Also read `paused.detail` in
to retry them and check again: ticks visit a paused base, and a base that stores withdrawals,
through its open dashboard issue (see [State](#state)). Also read `paused.detail` in
that file for the words `Kept nowhere (both lists full)`, and the history
(`npm run queue -- history --base <base> --limit 200`, every stored event, each cut at 200
characters there; the whole detail is in `history[]` of the same file) for `error` events that
@@ -1797,7 +1842,7 @@ and the event.
What you see: the dashboard banner "⏸️ Queue paused — reason `base-red` since <time>", its detail
(`all <n> failure(s) also fail on base <sha12>`, or `<batchId> failed like the bare base`), the
banner line "Resumes automatically when the base tip moves and turns green.", a `base-red-paused`
banner line "Resumes automatically when the base tip moves.", a `base-red-paused`
event and, when a `🔴 Release branch not green: <base>` issue is open, a comment on it that starts "🔴
omni-queue paused `<base>`". The batch is innocent and stays queued. Fix the base through its own
`fix/release-vX.Y.Z-basereds` PR (AGENTS.md → "Base-green check"); the pause lifts by itself when
@@ -1818,8 +1863,13 @@ If the failure turns out to be flaky and the tip will not move soon, resume by h
What you see, in live mode, on the repair PR: a comment that starts "🔴 omni-queue: this PR is in the
base-repair lane of `<base>`, but its candidate failed with the same failures as the base". The two
ways out are in [The two lanes](#the-two-lanes-lanes). An `infra` or `error` pause reached while a
repair is in flight replaces the `base-red` pause, as for any batch. After the resume the repair is
ways out are in [The two lanes](#the-two-lanes-lanes). A repair whose CI fails for infrastructure
after its retries is dropped and the `base-red` pause stays (an `error` event that ends
`dropped; the base-red pause stays`); the lane composes the repair again, a new candidate, so an
Actions outage costs one candidate per `infraRetries` + 1 runs until it ends or the base moves, and
after 120 minutes the `base-red` pause is a stalled queue. An `error` pause reached while a repair is
in flight (a landing that cannot go on, a full withdrawal list) replaces the `base-red` pause, as for
any batch. After the resume the repair is
taken again; if it lands there is no `resumed` event `base repaired by #<n>`, because no `base-red`
pause was left; without a valid repair, the next ordinary batch on the red base is classified
base-red again from the recorded verdict. A lane label removed while its candidate is in CI does not
@@ -1841,9 +1891,13 @@ take the PR snapshot and run eligibility (dashboard, eligibility comments and la
delete stray candidate refs, judge the queue's health and update the dashboard. A pause of more than
120 minutes is a stalled queue (see [Health](#health)): one alert mentions the approvers. The pause
tick itself sends a batch at the landing stage back to `testing` and withdraws its land status (with
`merge-ff`, a batch whose candidate is already the base tip is recorded as landed instead). To stop
every base, use the kill switch, and cancel any running `omni-queue` run (see [Kill
switch](#kill-switch)).
`merge-ff`, a batch whose candidate is already the base tip is recorded as landed instead; when that
batch was a segment of a bisection, the next segment is not composed under the pause: the bisection
ends there, and its PRs keep their labels and come back through the FIFO after the resume). The
reason is stored on one line, cut at 200 characters, with `Kept nowhere` written `Kept-nowhere`. A
pause or a resume needs only the right to dispatch the workflow (no approver check; see
[CLI](#cli)). To stop every base, use the kill switch, and cancel any running `omni-queue` run (see
[Kill switch](#kill-switch)).
### Recomposition when the base moves
@@ -1910,9 +1964,15 @@ usually only one of them speaks for a cause; both can, when a comment that could
(`notified` in `stalled` once it is posted);
- the watchdog issue, `omni-queue: the queue is not making progress`
(`scripts/merge-queue/watchdog.mjs`, every 30 minutes): when a base with work (a candidate in
flight, a pause, or a labeled PR, drafts included) has saved no state for more than 90 minutes;
when an episode's comment has not been posted 30 minutes after the episode opened, on a base that
is not an orphan (no work and no save for 90 minutes); or when a state file cannot be read: it is
flight, a pause, a stored land-status withdrawal, or a labeled PR, drafts included) has saved no
state for more than 90 minutes; when a managed base that labeled PRs target has no state file at
all and the newest update of those PRs is more than 90 minutes old ("**Labelled pull requests, no
state file**": no tick ever completed or recorded a failure for it, as when the App token cannot
be minted); when the queue's config does not load or does not validate ("**The queue's config
cannot be loaded**", with its one-line reason; such a run judges nothing else, never closes the
issue, and mentions the approvers only when the raw JSON has an `approvers` array of valid
logins); when an episode's comment has not been posted 30 minutes after the episode opened, on a
base that is not an orphan (no work and no save for 90 minutes); or when a state file cannot be read: it is
not JSON, it fails `validateState`, or its document is valid but not where its base belongs. That
last case has two reasons: "the document is for base B, which the queue does not manage" (its own
file, or a name the store refuses), and "the document is for base B, which is stored in
@@ -1949,16 +2009,30 @@ watchdog's own client limits (`WATCHDOG_MAX_RETRIES`, `WATCHDOG_REQUEST_TIMEOUT_
seconds; a `Retry-After` longer than 15 seconds, or a rate-limit reset further away than 15
seconds, is not waited for: the read fails at once). The deadline
leaves the job's 5-minute limit about a minute for checkout and Node setup; only when those take
longer is the run killed before it prints its line. A deadline that fires during a write can
leave a comment without its state change, as a killed job could: the next run finds the issue
again by its title and marker.
longer is the run killed before it prints its line. Once the deadline has fired the run starts no
write, and a reopen or a close stops between its comment and its issue update: the next run finds
the issue again by its title and marker. The run itself is still pending then; the workflow's
script exits the process at once.
Neither channel speaks in two cases. An invalid `config/merge-queue/config.json` on the default
branch stops the ticks, the approval job and the watchdog before they do anything (see
[Invalid configuration on the default branch](#invalid-configuration-on-the-default-branch)): only
red runs show it. A base whose only work is stored withdrawals is not visited by the ticks, and the
watchdog does not count those withdrawals as work either (see the known limits in
[Red candidates](#red-candidates-classification-base-check-bisection)).
What the dashboard alert covers, by its rules (see [Health](#health)): a pause of more than 120
minutes, an `error` or `tree-divergence` pause at once, a candidate in flight too long, eligible PRs
with nothing started, three failed ticks in a row, a land-status withdrawal GitHub has refused for
more than an hour, and three landings in a row without a verdict. What the watchdog adds: a base
whose ticks stopped saving, a base with labeled PRs and no state file, an undelivered alert, an
unreadable state file and an invalid config.
What still tells nobody (each shows on the dashboard or in the history only):
- a labeled PR that waits on `pr-checks-not-green`, `approval-missing`, `label-not-by-approver` or
any other eligibility reason: it is not eligible, so the waiting rule does not count it;
- a check run on a candidate that is no job of its push runs: one `error` event, nothing more;
- a `would-land` event that says `live mode would pause: …` or `… could not run (…)` in shadow
mode;
- a `base-red` pause younger than 120 minutes, and a release freeze that holds a green batch;
- the watchdog's own workflow failing before its script runs (a broken workflow file, a lost
runner): only its red scheduled runs show it;
- a person closing a dashboard issue: nothing is told until the base's state stops moving, then the
watchdog reports it after 90 minutes.
### The queue stalled
@@ -1975,10 +2049,19 @@ its cause:
- `eligible PRs waiting with nothing in flight since …`: ticks complete but compose nothing; read
the latest `omni-queue` runs (the decision lines say why) and the `error` events;
- `<n> ticks in a row failed, the first at …`: read the `error` events
(`npm run queue -- history --base <base>`) and the failed `omni-queue` runs.
(`npm run queue -- history --base <base>`) and the failed `omni-queue` runs;
- `paused (reason <reason>) since …: a pause for this reason does not lift by itself` (`error` or
`tree-divergence`): read the pause detail and its entry in Incidents;
- `stored land-status withdrawals not written for more than 60 min (<n> in landWithdrawals, <m> in landWithdrawalsOverflow): the oldest, <sha12> of batch <batchId>, refused since …`:
see [A withdrawal GitHub refused, or one refused for a
day](#a-withdrawal-github-refused-or-one-refused-for-a-day);
- `<n> landings in a row ended without a verdict, the first at …, none landed since: the landing keeps failing`:
read the `error` events of the failed ticks (the error each landing threw) and the base's history
for what keeps moving it.
The episode ends by itself at the first completed tick that finds none of these; nothing needs to
be closed by hand. The watchdog issue closes itself once none of its signals fires.
The episode ends by itself at the first completed tick that finds none of these (an episode that
failed ticks opened, after two completed ticks in a row); nothing needs to be closed by hand. The
watchdog issue closes itself once none of its signals fires.
### Dashboard stopped updating
@@ -1999,10 +2082,8 @@ ruleset applied, disable it first); an invalid `config/merge-queue/config.json`
discovery (red runs with `setup failed` or `base discovery failed`, see
[Invalid configuration on the default branch](#invalid-configuration-on-the-default-branch) and
[A tick fails before any base](#a-tick-fails-before-any-base-base-discovery-failed)); a queue run
that is pending forever (only one tick runs at a time, by design). A base whose only work is
stored withdrawals is not visited at all, so its dashboard stops changing although nothing is
wrong with the ticks (see the known limits in
[Red candidates](#red-candidates-classification-base-check-bisection)).
that is pending forever (only one tick runs at a time, by design); a closed dashboard issue, on a
base whose only work is not a labeled PR (see [State](#state)).
### Dequeued PR
@@ -2071,7 +2152,8 @@ that is not a success, a moved base or a tree divergence: `rejected` (the push w
ruleset, a protected-branch hook or another server-side rule, or the base was missing at the
landing's own read), `not-descendant` (a local check before any push: the stored base is not an
ancestor of the candidate) or, with `squash-api`, `merge-failed` (GitHub refused one PR's merge; the
PRs merged before it are on the base and named on the `error` event). The status written for the
PRs merged before it are on the base and named on the `error` event; a rate limit is not a refusal:
it is retried, see [Who lands](#who-lands-landingactor)). The status written for the
landing was withdrawn. Read the detail: for `rejected`, check the rules GitHub enforces on the base
(the landing-path check of `apply-ruleset.mjs --kind release` lists the rules of other rulesets that
block the landing actor, see [Ruleset](#ruleset)); for `merge-failed`, look at the PR GitHub
@@ -2090,32 +2172,30 @@ threw an error that is neither an authentication refusal nor a known refusal, or
the blessing. The status of the batch's commits was withdrawn. Read the last `error` events of
`npm run queue -- history --base <base>` (each cut at 200 characters there; the whole detail is in
`history[]` of `state/<sanitized-base>.json`) and the failed `omni-queue` runs. A GitHub rate limit
that outlasts three ticks ends here too: with `merge-ff` a rate limit is retried, not treated as a
refused token (with `squash-api` it pauses at once, see the known limits in [Red
candidates](#red-candidates-classification-base-check-bisection)). Once the cause is gone,
that outlasts three ticks ends here too: in both landing modes a rate limit is retried, not treated
as a refused token. Once the cause is gone,
`npm run queue -- resume <base>`: the pause dropped the batch, and after the resume its PRs are
composed again.
### A withdrawal GitHub refused, or one refused for a day
What you see: an `error` event whose detail reads
`<batchId>: could not withdraw <landing.statusContext> (<why>) on <sha12> (<error>)`, or, about a
day later, one that reads
`<batchId>: the withdrawal of <landing.statusContext> on <sha> (<list>) has failed 288 times since <time>; the queue keeps retrying it`.
`<batchId>: could not withdraw <landing.statusContext> (<why>) on <sha12> (<error>)`; an hour
later, a stalled alert whose reason starts `stored land-status withdrawals not written for more than 60 min`;
a day later, an `error` event that reads
`<batchId>: the withdrawal of <landing.statusContext> on <sha> (<list>) has been refused for more than 24 h, since <time>; the queue keeps retrying it`.
The dashboard and `npm run queue -- status` show the counts of the stored withdrawals.
What it means: GitHub refused to write the App's `failure` land status on a commit whose blessing
had to be withdrawn, so that commit may still carry the App's `success` and can be merged by hand
past the release ruleset. The refusal is stored and retried (see
[Who lands](#who-lands-landingactor)). The second detail means the `success` has stood for about a
day (`WITHDRAWAL_ALERT_ATTEMPTS`, 288 failed attempts).
[Who lands](#who-lands-landingactor)). The last detail means the `success` has stood for a day
(`WITHDRAWAL_ALERT_MINUTES`, counted from the entry's `since`).
What to do: find out why GitHub refuses the App's status writes, as in step 1 of
[Paused: landWithdrawals is full](#paused-landwithdrawals-is-full). Then make sure ticks visit the
base: a base whose only work is stored withdrawals is not visited (see the known limits in
[Red candidates](#red-candidates-classification-base-check-bisection)), so pause it
(`npm run queue -- pause <base>`) or dispatch a tick for it from the Actions tab, and wait until the
entry is gone from `state/<sanitized-base>.json`.
[Paused: landWithdrawals is full](#paused-landwithdrawals-is-full). The ticks keep visiting the base
while an entry is stored (through its open dashboard issue: do not close it) and retry the entry on
every run; wait until it is gone from `state/<sanitized-base>.json`. Nothing else needs a person.
### Paused: landWithdrawals is full
@@ -2141,8 +2221,10 @@ to that pause's detail. A later pause (an operator's `queue pause`, for one) kee
`Kept nowhere (both lists full)` name of the pause it replaces. A `base-red` pause is replaced by
this one (its detail says so), since the base-repair lane is blessed under it and it lifts by
itself. The new detail names that pause's issue, in the clause
`; the base was also paused as base-red (issue #<n>): <its detail>`. When a tick that completes
replaces it, one comment on its base-red issue says the base will not resume by itself (best effort:
`; the base was also paused as base-red (issue #<n>): <its detail>`. A tick that loads a `base-red`
pause together with entries in `landWithdrawalsOverflow` (a hand-edited state) replaces it the same
way, at its start. When a tick that completes replaces it, one comment on its base-red issue says
the base will not resume by itself (best effort:
a comment GitHub refuses is only a line in the run's log); when the failure path of a tick that
threw replaces it, no comment is posted. The dashboard and `npm run queue -- status` count the
entries of each list while either is not empty. Check, in order:
@@ -2194,14 +2276,17 @@ characters) and `npm run queue -- status` (200) cut long details. That commit's
What you see: red `omni-queue` runs whose step "Run one queue tick" prints
`omni-queue: setup failed: invalid omni-queue config <path>:` and one line per error; red
`Approval` jobs that print `omni-queue approval failed: invalid omni-queue config …`; red
`omni-queue watchdog` runs that print `omni-queue watchdog: invalid omni-queue config …`. A file
that is not JSON prints the JSON parser's message after the same prefixes instead.
`omni-queue watchdog` runs that print
`omni-queue watchdog: the queue's config cannot be loaded: invalid omni-queue config …`, and the
watchdog issue `omni-queue: the queue is not making progress` with the section "**The queue's config
cannot be loaded**" and the same reason on one line. A file that is not JSON prints the JSON
parser's message after the same prefixes instead.
What it means: `config/merge-queue/config.json` on the default branch fails `validateConfig`. Every
tick stops before it loads any state: no `error` event, no `failedTicks`, no stalled alert, no
dashboard change. The approval job writes no approval status, and the watchdog stops at its own
config load, so no watchdog issue opens. The red runs are the only signal (a defect: nobody is
told).
dashboard change. The approval job writes no approval status. The watchdog opens (or keeps) its
issue with the reason, mentioning the approvers only when the raw JSON still has an `approvers`
array of valid logins; it judges nothing else in that run, never closes the issue, and exits 1.
What to do: fix the file through a pull request into the default branch (the errors name each key).
The queue cannot land that pull request and a `bless` cannot run either (both load the same file),
@@ -2214,8 +2299,8 @@ What you see: a red `omni-queue` run whose step "Run one queue tick" prints
`omni-queue: base discovery failed: <message>`.
What it means: a tick without a base could not list the open pull requests or the dashboard issues
(`discoverBases`), so it visited no base and recorded nothing. The next tick, at most five minutes
later on the schedule, tries again. When it keeps failing, no state is saved: after 90 minutes the
(`discoverBases`), so it visited no base and recorded nothing. The next scheduled tick tries again
(the schedule asks for one every five minutes; GitHub often starts scheduled runs later). When it keeps failing, no state is saved: after 90 minutes the
watchdog reports each base with work (see [Who is told, and when](#who-is-told-and-when)). Read the
message (an API outage, a revoked App permission), and check the App's installation.
@@ -2237,6 +2322,34 @@ What you see, from `npm run queue -- pause|resume|bless`:
- `<base>: resume refused: …`: see [Paused: landWithdrawals is
full](#paused-landwithdrawals-is-full).
### Messages of a refused dispatch or a failed step
- `omni-queue: "<base>" is not a base the queue manages (config.bases: <pattern>): nothing loaded or written`
(a dispatch whose `base` does not match `bases`, or a name the store refuses), and the CLI's
`<base> is not a queue-managed base (config.bases <pattern>)`: check the base name. Nothing was
written.
- A raw dispatch from the Actions tab with inputs the tick refuses, printed as
`omni-queue: <message>` with nothing loaded: `--action <pause|resume|bless> requires --base`,
`--action bless requires --sha`, `--sha must be a full 40-hex commit SHA (lower case)`,
`--action bless requires --reason`, `--sha is only valid with --action bless`. Dispatch again with
the inputs the message names (the CLI checks the same ones before dispatching).
- A red run whose step "Resolve the bot identity" says `unexpected App slug` (the token step gave
an App slug that is not lowercase letters, digits and dashes) or
`could not resolve the user id of <slug>[bot]` (`GET /users/<slug>[bot]` failed): check the App
token step and the App's installation; no tick ran.
- `state of <base> is invalid: <fields>` from the CLI: the state document fails `validateState`
(see [State](#state)); its ticks fail too.
- `omni-queue <base>: the failed tick's record was not applied: <message>` in the log of a failed
run: `main()` recorded the tick's `error` event without what the tick left for the state, so the
withdrawals GitHub refused during that tick are not stored. Check the land status of the commits
that tick blessed (its log names them) and withdraw by hand where needed (the last resort of
[Paused: landWithdrawals is full](#paused-landwithdrawals-is-full)).
- `state of <base> changed during the tick` as a failed tick's error: three ticks in a row lost the
compare-and-swap of the state save (the log shows
`omni-queue <base>: state changed during the tick, redoing it (<n>/3)` before); `main()` records
the error. Something else writes `omni-queue-state` (a second writer, an edit by hand while the
queue is enabled): find it.
### Labels with no effect
If labeled PRs stay `approval-missing` and no `omni-queue/approval`
@@ -2321,7 +2434,9 @@ the cut before declaring the rollback done (see [Kill switch](#kill-switch)). Th
- delete the `omni-queue release protection: <base>` rulesets in Settings → Rules → Rulesets
(while they exist, nothing reaches those branches without the land status) and restore "Require
PR on default branch" if it was removed;
PR on default branch" if it was removed. When the code goes back too, do this when
[Rolling the code back](#rolling-the-code-back) says: its step 3, or its step 6 when step 1
blessed the rollback's head;
- re-enable the Mergify queue rules;
- delete the `omni-queue refs protection` ruleset if the state branch or the candidate branches
have to be changed by hand. Leftover `omni-queue/**` candidate branches can be deleted once
@@ -2364,27 +2479,37 @@ In this order; each step works in the state the step before it leaves:
`config/merge-queue/config.json`, and deletes `.github/workflows/omni-queue-watchdog.yml`.
Without the release rulesets it merges like any pull request; with them, only through the
blessing of step 1.
6. Right after that merge, when step 3 left the release rulesets in place (step 1 blessed the
rollback's head): with OMNI_QUEUE_ENABLED `false` nothing can write the land status any more
(the rulesets have no bypass actor), so those release branches accept no merge and no push at
all. Either delete the `omni-queue release protection: <base>` rulesets now (Settings → Rules →
Rulesets) and restore "Require PR on default branch" if it was removed, or set
OMNI_QUEUE_ENABLED back to `true` so that the rolled-back queue runs and blesses again. Do one
of the two before anything else has to reach a release branch.
Rolling forward again goes the same way: the release rulesets absent (re-apply them afterwards,
[Cut-over and rollback](#cut-over-and-rollback), step 6) or the head blessed while a queue runs,
and the state edited first, by the column "Before rolling forward again".
What the re-runs, the adaptive batch, the lanes and the health checks added to the state, and what
the code before them does with each:
What the re-runs, the adaptive batch, the lanes, the health checks and the final fix wave added to
the state, and what the code before them does with each:
| Added | The code before it | Before rolling back | Before rolling forward again |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `flaky-rerun` events | refuses the document | remove them | nothing |
| `queue-stalled` events | refuses the document | remove them | nothing |
| `landWithdrawalsOverflow` | carries it along, never retries its entries: a blessing stored there stays until the roll-forward | empty it in step 1, while the paused base's ticks still retry it; once the variable is `false` nothing retries it: then move its entries into `landWithdrawals` (when that list has room) in the commit of step 4 | nothing |
| `inFlight.landAttempts` at the stage `testing` (kept by a failed tick) | ignores it, and replaces it at the batch's next landing | nothing | nothing |
| `failedTicks`, `failedSince` | ignores them and never deletes them | nothing | remove them: otherwise the first failed tick counts the old ones (the older code saves hourly too, so the 90-minute gap that restarts the count does not apply) |
| `stalled` (`since`, `reasons`, `openedAt`, `notified`) | ignores it | nothing | remove it when it has no `notified: true`: the first new tick would post its old alert (a paused base is stalled after 120 minutes), and the watchdog would report it at once |
| `batchWindow` | ignores it; its batches take up to `batch.maxSize` | nothing | nothing |
| `inFlight.rerun` | ignores it | nothing | nothing |
| `inFlight.lane` (and `inFlight.pendingRed.lane`) | processes a lane batch as an ordinary batch; under a `base-red` pause it leaves it as is | nothing | nothing |
| `inFlight.freezeHold` | carries it along unchanged | nothing | nothing: the first tick that does not hold the batch closes it |
| `repairTried` | ignores it, never prunes it, never composes a repair alone | nothing | nothing: the first tick prunes it |
| Added | The code before it | Before rolling back | Before rolling forward again |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `flaky-rerun` events | refuses the document | remove them | nothing |
| `queue-stalled` events | refuses the document | remove them | nothing |
| `landWithdrawalsOverflow` | carries it along, never retries its entries: a blessing stored there stays until the roll-forward | nothing more than step 1, which waits until it is empty (the paused base's ticks retry it); if entries are left once the variable is `false`, nothing retries them: move them into `landWithdrawals` (when that list has room) in the commit of step 4 | nothing |
| `inFlight.landAttempts` at the stage `testing` (kept by a failed tick) | ignores it, and replaces it at the batch's next landing | nothing | nothing |
| `failedTicks`, `failedSince` | ignores them and never deletes them | nothing | remove them: otherwise the first failed tick counts the old ones (the older code saves hourly too, so the 90-minute gap that restarts the count does not apply) |
| `stalled` (`since`, `reasons`, `openedAt`, `notified`) | ignores it | nothing | remove it when it has no `notified: true`: the first new tick would post its old alert (a paused base is stalled after 120 minutes), and the watchdog would report it at once |
| `batchWindow` | ignores it; its batches take up to `batch.maxSize` | nothing | nothing |
| `inFlight.rerun` | ignores it | nothing | nothing |
| `inFlight.lane` (and `inFlight.pendingRed.lane`) | processes a lane batch as an ordinary batch; under a `base-red` pause it leaves it as is | nothing | nothing |
| `inFlight.freezeHold` | carries it along unchanged | nothing | nothing: the first tick that does not hold the batch closes it |
| `repairTried` | ignores it, never prunes it, never composes a repair alone | nothing | nothing: the first tick prunes it |
| `landingsWithoutVerdict` | ignores it, carries it along and never deletes it | nothing | remove it: the first new tick would count on from the old run (a stall at 3) |
| `stalled.quietTicks` | ignores it | nothing | nothing (the row of `stalled` applies) |
| stored withdrawals as work | does not visit a base whose only work is a stored withdrawal, nor does its watchdog judge it | nothing more than step 1, which empties both lists | nothing |
`pr-checks-not-green` is never stored, so it needs nothing.