fix(ci): omni-queue landing count belongs to the candidate; full lists never hide a commit

Task 13b, fix round 2, rulings W1, W2, W3, W4 (code items) and W5.

W1: when both lists are full and the base is already paused for a kept
reason, "Kept nowhere (both lists full): <sha> (<batchId>)" is appended to
that pause's detail (reason and since untouched), so paused.detail names
every commit kept nowhere. validateState has no bound on paused.detail;
the error event's detail is cut at 500 characters, so it names the
commits that fit and "and N more, named in paused.detail" (namedWithin).

W2: only manual, error, infra and tree-divergence pauses are kept
(KEPT_PAUSE_REASONS: none lifts by itself, nothing is blessed under one).
A base-red pause is replaced by the full-list pause, whose detail says
the base was also paused as base-red (the repair lane blesses under it,
maybeAutoResume lifts it when the tip moves).

W3: inFlight.landAttempts belongs to the candidate. Checks back to
pending, a re-run, a release-freeze hold, a paused base leaving it as is,
an infra retry (same SHA), the failure path and the full-list pause keep
it (written out: a stored landing without the field counts as 1); only a
new candidate and an operator's pause or resume (handleAction) start it
again. The limit is checked in live mode only: a shadow tick never pauses
on a carried count. Contract comment, tick header and runbook updated.

W4: only a NEW entry of the overflow is the full-list condition
(isFullListRefusal; storeWithdrawalIn says whether the entry is new). A
read or a write of a stored withdrawal that fails with a rate limit
(isRateLimitError) stops the retries of the tick; the rest of both lists
waits. One error event when an entry's attempts reach
WITHDRAWAL_ALERT_ATTEMPTS (288), at the crossing only, from the retry,
the in-tick store and the failure path. recordEarlierLanding reads the
base tip twice before falling back.

W5 runbook: the withdrawal is written once, by the pass right after the
batch in flight; the attempts sentence; the full-list incident (a NEW
overflow entry; which pauses are kept; base-red replaced; the counts on
the dashboard and in status); where a commit kept nowhere is named and
how long each view is; step 2 of a mode change reads paused.detail and
the history with --limit 200; the last-resort hand edit says what
disabling the queue-refs ruleset does to other live bases and that the
kill switch avoids it; an unknown commit's statuses answer an empty list.

New tests (tick.test.mjs), failed on 021ff70 unless marked:
- "W1 (re-review RR-8): a base paused by hand, both lists full: the
  commit kept nowhere is named in that pause's detail, its reason and
  since untouched"
- "W1: more commits kept nowhere than an event detail holds: all in
  paused.detail, the event names what fits and how many more"
- "W2: a base-red pause with a repair in flight is replaced by the
  full-list pause: nothing is blessed under it and it does not lift when
  the tip moves"
- "W3 (re-review RR-2): landings that throw, alternating with checks back
  to pending, pause at the third landing without a verdict"
- "W3 (re-review RR-10): a shadow tick never pauses on a kept count"
- "W3: a recomposition starts the count again" (passed at once: pins it)
- "W3: an operator's pause, and an operator's resume, start the count
  again"
- "W4 (re-review RR-9a): a refusal of a SHA already in the overflow, with
  room in the main list, is not a full list"
- "W4: stored retries stop at a rate limit: the rest of both lists waits
  for the next tick"
- "W4: one error event when a stored withdrawal reaches
  WITHDRAWAL_ALERT_ATTEMPTS, not on the ticks after"
- "W4: the tip read of recordEarlierLanding is tried twice before falling
  back"

Re-aimed to W3 (each asserted that landAttempts was gone; each now
asserts the candidate's count is kept):
- "a red batch at the landing stage leaves it before its failed jobs are
  re-run"
- "a red batch at the landing stage loses its land status on the first red
  tick, while it waits"
- "a landing-stage batch whose checks went back to pending is withdrawn
  and re-blessed when green" (its landing marker is now "(attempt 2)")
- "live → shadow on a paused base: a batch left at the landing stage loses
  its status too"
- "a batch at the landing stage held by a release-freeze: its land status
  turns failure; blessed again when the freeze lifts and it lands" (1
  when blessed, none otherwise)
- "M1 probe: a landing-stage batch whose base moved and whose
  recomposition push throws six times: withdrawn by the first tick, not
  stored as landing after it" (count 1 kept)
- round 1's "V1: a freeze hold between two failing landings still resets
  the count" is now "W3 (was V1): a freeze hold between two failing
  landings keeps the count" (kept through the hold, then 2)
Re-aimed to W4: "V5: the failure path leaves a batch whose candidate is
the base tip at the landing stage, with no write; the next tick records
it" now makes both of recordEarlierLanding's reads fail (it read once).
This commit is contained in:
diegosouzapw
2026-11-16 16:21:46 -03:00
parent ff07520472
commit 44a22145ab
4 changed files with 588 additions and 144 deletions

View File

@@ -190,27 +190,32 @@ shadow tick that meets such a batch records it the same way; shadow mode lands n
records a landing a live tick made.
A withdrawal GitHub refuses while the landing withdraws its own blessing (the landing ended with a
verdict other than landed) is tried once more by the end-of-tick pass above; other refusals wait
for the next tick. Every refused write is recorded with an `error` event, detail
verdict other than landed) is tried once more by the pass that runs right after the batch in
flight is handled (below); other refusals wait for the next tick. Every refused write is recorded with an `error` event, detail
`<batch>: could not withdraw <landing.statusContext> (<why>) on <sha12> (<error>)`, and stored in
the state field `landWithdrawals` (see [State](#state)). Every later tick, on a paused base too,
retries the stored withdrawals before it handles the batch in flight: it reads the land statuses
of each stored commit, writes `failure` where the same rule finds a `success` (the App's newest,
or the newest of the App's and those without a readable creator), and drops the entry once that
write succeeds or neither is `success` any more (or there is no status). A refused retry keeps
the entry and counts an attempt, and so does a failed read: `attempts` is the number of failed
attempts so far (a read that failed or a write that was refused). No count drops an entry: the
queue never gives up on withdrawing a blessing. A commit the tick blesses for landing, or an
approver blesses by hand, leaves the list.
write succeeds or neither is `success` any more (or there is no status: GitHub answers an empty
list for a commit it does not know, so such an entry leaves at its next retry). A refused retry
keeps the entry and counts an attempt, and so does a failed read: `attempts` is the number of
failed attempts so far (a read that failed or a write that was refused). No count drops an entry:
the queue never gives up on withdrawing a blessing. When an entry's `attempts` reach 288
(`WITHDRAWAL_ALERT_ATTEMPTS` in `scripts/merge-queue/tick.mjs`, about a day of ticks), one `error`
event names the commit and its list and says the queue keeps retrying. A read or a write that
fails with a rate limit stops the retries of that tick: the rest of both lists waits for the next
tick. A commit the tick blesses for landing, or an approver blesses by hand, leaves the list.
Shadow mode retries the stored withdrawals too, and withdraws the status of a batch stored at the
landing stage, so a switch from live to shadow strands no blessing (see
[Before a mode change or a stop](#before-a-mode-change-or-a-stop)); a queue that was never live
has nothing to withdraw.
A batch the tick loaded at the landing stage and no longer keeps there loses its blessing twice
over: right after the batch in flight is handled, before the queue snapshot and any composition
(which could hang until the job's timeout), and again at the end of the tick for what the rest of
the tick changed (the same read, once per batch).
A batch the tick loaded at the landing stage and no longer keeps there loses its blessing once,
right after the batch in flight is handled, before the queue snapshot and any composition (which
could hang until the job's timeout). The end of the tick runs the same pass again, which writes
only for a batch that left the landing stage later in the tick (a pause that a full withdrawal
list caused, for example).
A tick that throws (any error but a lost compare-and-swap; never in a dry-run) saves no state of
its own after the error; what it saved before (the landing stage and its count, written before a
@@ -235,11 +240,18 @@ The list holds at most 20 entries (`LAND_WITHDRAWALS_LIMIT` in
is stored in the state field `landWithdrawalsOverflow` (same entries, at most 20 more), and both
lists are retried by the same code on every tick, in every mode. A full main list means GitHub is
refusing the App's status writes, so a base that is not paused yet is paused with reason `error`; the
detail starts with `landWithdrawals is full` and says how many entries each list holds. A base
already paused keeps its own pause (reason, detail, time); an `error` event records the full list.
Only a refusal that fits in neither list is kept nowhere: its commit is named in full in the pause
detail (a pause of this kind gets it appended) and in that `error` event (see
[Incidents](#incidents)).
detail starts with `landWithdrawals is full` and says how many entries each list holds. Only a NEW
entry in the overflow is that condition: a refusal of a commit the overflow already holds is not.
A base already paused for a reason that neither lifts by itself nor blesses (`manual`, `error`,
`infra`, `tree-divergence`) keeps its pause (reason, time); an `error` event records the full
list. A `base-red` pause is replaced by the full-list pause, whose detail says the base was also
paused as base-red: under a base-red pause the base-repair lane is still blessed, and the pause
lifts by itself when the base tip moves. Only a refusal that fits in neither list is kept nowhere:
its commit is named in full, after the words `Kept nowhere (both lists full)`, in `paused.detail`
whatever the pause (appended to a kept pause's detail), and in that `error` event, whose detail is
cut at 500 characters: it names the commits that fit and how many more `paused.detail` holds
(see [Incidents](#incidents)). The state has no bound on `paused.detail`; the dashboard (300
characters) and `npm run queue -- status` (200) show it cut.
In live mode:
@@ -260,8 +272,11 @@ In live mode:
counted in `inFlight.landAttempts` and saved before its blessing, so a tick that died after the
blessing counts too. After `LANDING_ATTEMPTS_LIMIT` (3, in `scripts/merge-queue/tick.mjs`)
attempts in a row without a verdict the batch is not blessed again and the base is paused with
reason `error` (see [Incidents](#incidents)). The count goes with the landing stage: a held batch,
a manual pause or an infra retry starts it again.
reason `error` (see [Incidents](#incidents)). The count belongs to the candidate, not to the
stage: checks back to pending, a re-run, a release-freeze hold, an infra retry and a failed tick
keep it; only a new candidate (a recomposition, a new batch) and an operator's pause or resume
start it again. Shadow mode never checks the limit (it blesses nothing), so a count carried from
live mode never pauses a shadow tick.
- **A per-PR merge call of `squash-api` that fails** (a rate limit included) is not retried: the
error is caught inside `landSquashApi` and is a verdict, `merge-failed`, so the base is paused
with reason `error`. The rate-limit rule above decides the token hint here too: with
@@ -515,11 +530,11 @@ candidate in flight (refs, SHAs, PRs, stage, attempts, bisection), the last base
shadow bookkeeping, counters and the last 200 history events. Three fields belong to the land status
(see [Who lands](#who-lands-landingactor)):
- `inFlight.landAttempts`: the number of ticks in a row that blessed the batch without reaching a
verdict. Present at the stage `landing`, and at the stage `testing` only when a failed tick sent
the batch back there (the failure path keeps the count); a hold, checks back to pending, a re-run,
a pause, an infra retry or a recomposition deletes it. At `LANDING_ATTEMPTS_LIMIT` (3) the queue
pauses instead of blessing it again.
- `inFlight.landAttempts`: the number of ticks that blessed the candidate without reaching a
verdict. It belongs to the candidate, at either stage: a hold, checks back to pending, a re-run,
an infra retry and a failed tick keep it; a new candidate (a recomposition, a new batch) and an
operator's pause or resume start it again. At `LANDING_ATTEMPTS_LIMIT` (3) a live tick pauses
instead of blessing it again; shadow mode never checks it.
- `landWithdrawals`: present only when it is not empty. Each entry is a commit whose `failure` land
status GitHub refused to write (`sha`, `batchId`, `candidateSha`, `why`, `since`, `attempts`),
oldest first, one per SHA, at most 20. Later ticks retry them.
@@ -780,10 +795,11 @@ queue to shadow (or shadow to live), turning the kill switch off or rolling back
`stage <stage>` for the batch in flight) and that both `landWithdrawals` and
`landWithdrawalsOverflow` are empty: the fields are absent from `state/<sanitized-base>.json` on
`omni-queue-state` (an empty list leaves no field). While entries remain, wait for the next ticks
to retry them and check again. Also read the `error` events of the history
(`npm run queue -- history --base <base>`) for one that starts with `landWithdrawals is full` and
says `kept nowhere`: each commit it names may still carry the App's `success` (see
[Incidents](#incidents), "Paused: landWithdrawals is full").
to retry them and check again. Also read `paused.detail` in that file for the words
`Kept nowhere (both lists full)`, and the history
(`npm run queue -- history --base <base> --limit 200`, every stored event) for `error` events
that start with `landWithdrawals is full`: each commit named after those words may still carry
the App's `success` (see [Incidents](#incidents), "Paused: landWithdrawals is full").
3. Only then make the change. After a mode change, resume each base
(`npm run queue -- resume <base>`).
@@ -1519,12 +1535,17 @@ refused token. Once the cause is gone, `npm run queue -- resume <base>` recompos
**Paused: landWithdrawals is full.** A pause detail that starts with
`landWithdrawals is full (<n> of 20; landWithdrawalsOverflow holds <m> of 20)` means GitHub refused
to write the App's `failure` land status while 20 refused withdrawals were already stored: GitHub
is refusing the App's status writes, so the queue stopped blessing. Every refused withdrawal is
kept, in `landWithdrawals` or in `landWithdrawalsOverflow`, and both lists are retried on every
tick of the paused base. A base that was already paused (a manual pause before a mode change, for
example) keeps its own pause: the same condition is then only an `error` event that starts with
`landWithdrawals is full`. Check, in order:
to write the App's `failure` land status on a commit that had to become a new entry of the
overflow, because 20 refused withdrawals were already stored: GitHub is refusing the App's status
writes, so the queue stopped blessing. Every refused withdrawal is kept, in `landWithdrawals` or in
`landWithdrawalsOverflow`, and both lists are retried on every tick of the paused base. A base that
was already paused for a reason that neither lifts by itself nor blesses (`manual`, `error`,
`infra`, `tree-divergence`; a manual pause before a mode change, for example) keeps its own pause:
the same condition is then an `error` event that starts with `landWithdrawals is full`, plus, for a
commit kept nowhere, its name appended to that pause's detail. A `base-red` pause is replaced by
this one (its detail says so), since the base-repair lane is blessed under it and it lifts by
itself. The dashboard and `npm run queue -- status` count the entries of each list while either
is not empty. Check, in order:
1. Why GitHub refuses the App's status writes: the run's log names the error of each refused
write, and so does an `error` event (`<batchId>: could not withdraw …`, or the detail of the
@@ -1540,14 +1561,18 @@ example) keeps its own pause: the same condition is then only an `error` event t
The last resort: when both lists were full too, a refused withdrawal is kept nowhere. Its commit is
then named in full, as `<sha> (<batchId>)` after the words `Kept nowhere (both lists full)`, in
`paused.detail` of `state/<sanitized-base>.json` and in the `sha` and detail of that `error`
event. Read it in the state file itself: the dashboard and `npm run queue -- status` cut
long details. That commit's App status may stay `success` and nothing retries it; a person has to
`paused.detail` of `state/<sanitized-base>.json`, whatever the pause, and in the `sha` and detail of
that `error` event (cut at 500 characters: it names the commits that fit and how many more
`paused.detail` holds). Read `paused.detail` in the state file itself: the dashboard (300
characters) and `npm run queue -- status` (200) cut long details. That commit's App status may stay `success` and nothing retries it; a person has to
put it back in a list, once a list has room (after step 2):
1. The base is already paused (this pause); keep it paused.
2. Disable the `omni-queue refs protection` ruleset (Settings → Rules → Rulesets), so an ordinary
commit can change `omni-queue-state`.
commit can change `omni-queue-state`. While it is disabled, any other live base whose tick
tries to bless pauses with `queue refs are not protected (…)` (see
[Before the land status is written](#before-the-land-status-is-written)); turning the kill switch
off first (see [Kill switch](#kill-switch)) avoids that, and you turn it on again after step 4.
3. In `state/<sanitized-base>.json`, append to `landWithdrawals` (or to `landWithdrawalsOverflow`;
at most 20 entries in each, and a SHA in one list only) one entry per named commit:
`{"sha": "<40-hex sha>", "batchId": "<batchId>", "candidateSha": "<40-hex sha>", "why": "left the landing stage", "since": "<ISO 8601 time, e.g. 2026-11-16T12:00:00.000Z>", "attempts": 1}`,

View File

@@ -193,14 +193,14 @@ const SAFE_SEGMENT_RE = /^[A-Za-z0-9._-]+$/;
* timeouts, set only while that holds for EVERY run (=
* attempts); absent otherwise, and in documents written before
* it existed (= first run not known to be a timeout)
* @property {number} [landAttempts] ticks that blessed this batch for landing and reached no
* verdict, counted (and saved) before each blessing. Set at the
* landing stage, and kept at stage "testing" only when main()'s
* failure path sent the batch back there (tick amendFailedTick);
* deleted by every other exit from the landing stage (hold,
* pending checks, re-run, pause, infra retry, recomposition).
* Absent in documents written before it existed (= 1 at the
* landing stage). See tick LANDING_ATTEMPTS_LIMIT
* @property {number} [landAttempts] ticks that blessed this candidate for landing and reached
* no verdict, counted (and saved) before each blessing. It
* belongs to the candidate, at either stage: kept through a
* hold, pending checks, a re-run, an infra retry and main()'s
* failure path; a new candidate (recomposition, new batch) has
* none, and an operator's pause or resume deletes it. Absent in
* documents written before it existed (= 1 at the landing
* stage). See tick LANDING_ATTEMPTS_LIMIT (live mode only)
* @property {RerunRecord} [rerun] the failed jobs of this candidate were re-run once (tick
* onRed); absent before that, in documents written before it
* existed, and on an infra retry (a new ref)

View File

@@ -78,8 +78,10 @@
// any later tick that drops, recomposes or pauses that batch turns the App's `success` land
// status on its commits into `failure` (withdrawAbandoned; only statuses created by the App
// count, as for the release ruleset). That save also counts the attempt
// (inFlight.landAttempts): after LANDING_ATTEMPTS_LIMIT attempts in a row without a verdict
// the batch is not blessed again and the queue pauses with reason "error". A batch HELD at
// (inFlight.landAttempts, the candidate's: kept at either stage, started again only by a new
// candidate and by an operator's pause or resume): after LANDING_ATTEMPTS_LIMIT attempts
// without a verdict a live tick does not bless the batch again and the queue pauses with
// reason "error" (shadow never checks it). A batch HELD at
// the landing stage (release-freeze, checks back to pending, manual pause) goes back to stage
// "testing" (leaveLandingStage), so the same tick withdraws its status and later held ticks
// read nothing; it is blessed again only right before it lands. A paused base that does not
@@ -89,9 +91,10 @@
// attempt withdrew its own blessing and keeps its count). A `failure` GitHub refuses to write
// is kept in state.landWithdrawals and retried by the next ticks (retryWithdrawals); when that
// list is full (LAND_WITHDRAWALS_LIMIT) nothing is dropped: the refusal goes to
// state.landWithdrawalsOverflow, retried alike, and a base that is not paused pauses with
// reason "error" (recordFullWithdrawals); only a refusal that fits in neither list is named,
// in full, in the pause detail and in an `error` event.
// state.landWithdrawalsOverflow, retried alike, and a base that is not paused (or paused as
// base-red, which blesses its repair lane and lifts by itself) pauses with reason "error"
// (recordFullWithdrawals); only a refusal that fits in neither list is named, in full, in
// paused.detail whatever the pause, and in an `error` event.
// Before landing (or composing a bisection segment) every PR is re-read and re-run through the
// eligibility rules: block labels, approval status on the head, protected approval and an open
// release-freeze (which holds the green batch in flight instead of landing it). Then the candidate
@@ -291,6 +294,14 @@ const AUTH_FAILURE_RE =
*/
const RATE_LIMIT_RE = /rate limit/i;
/**
* A GitHub error the client classifies as a rate limit (github.isRateLimited: a 429, or a
* primary or secondary rate limit, whose message says "rate limit"); retried no further this tick.
*/
function isRateLimitError(error) {
return error?.status === 429 || RATE_LIMIT_RE.test(String(error?.message ?? ""));
}
function isAuthFailure(error) {
if (error?.status === 401) return true;
const message = String(error?.message ?? "");
@@ -904,9 +915,9 @@ class Tick {
};
this.event("paused", { detail });
this.note(`paused manually: ${detail}`);
// Withdrawn by withdrawAbandoned; blessed again only when it lands after the resume. The
// landing count goes with the stage: the resume is the operator's fresh start.
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight);
// Withdrawn by withdrawAbandoned; blessed again only when it lands after the resume. An
// operator's pause starts the landing count again (as the resume does).
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight, { resetCount: true });
} else if (this.action === "resume") {
if (!this.s.paused) return this.note("resume: the queue was not paused");
const reason = String(this.reason ?? "");
@@ -922,6 +933,8 @@ class Tick {
}
const previous = this.s.paused.reason;
this.s.paused = null;
// The operator's fresh start: the landing count of the batch in flight starts again.
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight, { resetCount: true });
this.event("resumed", { detail: `manual resume (was ${previous})` });
this.note(`resumed (was ${previous})`);
}
@@ -1210,21 +1223,21 @@ class Tick {
* landing stage. Nothing is written here: this tick's withdrawAbandoned sees the batch is no
* longer kept at the landing stage and turns a `success` land status on its commits into
* `failure` (one read per commit, once: later ticks find it stored at "testing"). The landing
* count goes too (inFlight.landAttempts): a hold, checks back to pending, a re-run or a pause is
* a fresh start, also for a batch the failure path sent back to "testing" with its count kept
* (amendFailedTick). Otherwise a no-op at any other stage.
* count (inFlight.landAttempts) belongs to the candidate and stays (a hold, checks back to
* pending, a re-run, a paused base leaving it as is); only a new candidate and an operator's
* pause (`resetCount`) or resume start it again, the latter at any stage.
*/
leaveLandingStage(f) {
leaveLandingStage(f, { resetCount = false } = {}) {
if (f.stage !== "landing") {
if (f.landAttempts === undefined || this.s.inFlight !== f) return;
if (!resetCount || f.landAttempts === undefined || this.s.inFlight !== f) return;
const rest = { ...f };
delete rest.landAttempts;
this.s.inFlight = rest;
return this.note(`${f.batchId}: its landing attempts count starts again`);
}
const rest = { ...f };
delete rest.landAttempts;
this.s.inFlight = { ...rest, stage: "testing" };
const rest = { ...f, stage: "testing", landAttempts: storedLandAttempts(f) };
if (resetCount) delete rest.landAttempts;
this.s.inFlight = rest;
// A dry-run withdraws nothing (withdrawAbandoned returns early): say what would happen.
const goes = this.dryRun ? "would go" : "goes";
this.note(`${f.batchId}: held at the landing stage, back to testing (its land status ${goes})`);
@@ -1258,13 +1271,18 @@ class Tick {
const f = this.s.inFlight;
if (f?.stage !== "landing" || this.config.landing.mode !== "merge-ff") return;
let tip;
try {
tip = await this.gh.getBranchSha(this.base);
} catch (error) {
// Unknown: the batch is treated as not landed (withdrawn if this tick leaves it), the
// security-first choice of settleFailure too; the tick goes on (a pause is still applied).
this.note(`base tip unreadable, ${f.batchId} taken as not landed: ${error.message}`);
return;
for (let read = 1; ; read++) {
try {
tip = await this.gh.getBranchSha(this.base);
break;
} catch (error) {
// Two reads: the fall-back below can write `failure` on a tip that did land.
if (read < 2) continue;
// Unknown: the batch is treated as not landed (withdrawn if this tick leaves it), the
// security-first choice of settleFailure too; the tick goes on (a pause is still applied).
this.note(`base tip unreadable, ${f.batchId} taken as not landed: ${error.message}`);
return;
}
}
this.tipChecked = true;
if (tip !== f.candidateSha) return;
@@ -1377,8 +1395,9 @@ class Tick {
// Earlier ticks blessed this batch and none of them reached a verdict (stored at the landing
// stage, or sent back to testing by main()'s failure path with its count kept). After
// LANDING_ATTEMPTS_LIMIT of them it is not blessed again: pause and say so.
// Live only: shadow blesses nothing, so a count carried from live mode never pauses it.
const landAttempts = storedLandAttempts(f);
if (landAttempts >= LANDING_ATTEMPTS_LIMIT) {
if (live && landAttempts >= LANDING_ATTEMPTS_LIMIT) {
const lastError = this.s.history.findLast((e) => e.type === "error")?.detail || "unknown";
return this.pauseLanding(
f,
@@ -1764,8 +1783,9 @@ class Tick {
/**
* End of a tick: the withdrawals GitHub refused this tick while landWithdrawals was full
* (storeWithdrawal: kept in the overflow, or nowhere) are recorded by the pure
* recordFullWithdrawals: a base that is not paused pauses with reason `error`, a paused one keeps
* its pause, and an `error` event says which commits. A new pause leaves a landing stage (a
* recordFullWithdrawals: a base that is not paused (or paused as base-red) pauses with reason
* `error`, one paused for another reason keeps its reason (a commit kept nowhere is appended to
* its detail), and an `error` event says which commits. A new pause leaves a landing stage (a
* paused base holds no blessed batch), so withdrawAbandoned runs once more; a refusal there is
* recorded the same way.
*/
@@ -1815,16 +1835,18 @@ class Tick {
* Keep a `failure` GitHub refused to write on `sha` for the next ticks (retryWithdrawals), by
* the pure storeWithdrawalIn: one entry per SHA, in state.landWithdrawals while it has room,
* else in state.landWithdrawalsOverflow. Nothing is ever dropped. A refusal that did not go to
* the main list (overflow, or nowhere when both are full) is held in this.fullRefusals for the
* end of the tick (recordIfFull). Every refusal is also kept in this.refused for main()'s
* failure path.
* the main list as a new entry (overflow, or nowhere when both are full) is held in
* this.fullRefusals for the end of the tick (recordIfFull). Every refusal is also kept in
* this.refused for main()'s failure path.
*/
storeWithdrawal(f, sha, why) {
const refusal = { sha, batchId: f.batchId, candidateSha: f.candidateSha, why, at: this.now() };
this.refused.push(refusal);
const { state, list } = storeWithdrawalIn(this.s, refusal);
const result = storeWithdrawalIn(this.s, refusal);
const { state, list } = result;
for (const field of WITHDRAWAL_LISTS) this.setWithdrawalList(field, state[field] ?? []);
if (list === "landWithdrawals") return;
this.alertAttempts(result.entry, list);
if (!isFullListRefusal(result)) return;
this.fullRefusals.push({ ...refusal, list });
this.note(
`${f.batchId}: withdrawal of ${head12(sha)} ` +
@@ -1871,24 +1893,39 @@ class Tick {
if (this.dryRun) return;
for (const field of WITHDRAWAL_LISTS) {
const stored = this.s[field] ?? [];
if (stored.length) this.setWithdrawalList(field, await this.retryList(stored));
if (!stored.length) continue;
const { kept, stopped } = await this.retryList(stored, field);
this.setWithdrawalList(field, kept);
if (stopped) return; // a rate limit: the rest of both lists waits for the next tick
}
}
/** retryWithdrawals for one list: returns the entries kept (each with its attempts counted). */
async retryList(stored) {
/**
* retryWithdrawals for one list: returns the entries kept (each failed one with its attempts
* counted; an entry that reaches WITHDRAWAL_ALERT_ATTEMPTS records one `error` event). A read or
* a write that fails with a rate limit (isRateLimitError) stops the list there: the entries
* after it are kept as they are, and `stopped` tells the caller to leave the next list too.
*/
async retryList(stored, field) {
const { statusContext } = this.config.landing;
const kept = [];
for (const w of stored) {
const failed = (w, error) => {
const next = { ...w, attempts: w.attempts + 1 };
kept.push(next);
this.alertAttempts(next, field);
return isRateLimitError(error);
};
for (const [i, w] of stored.entries()) {
let newest;
try {
newest = await this.newestAppLandStatus(w.sha);
} catch (error) {
kept.push({ ...w, attempts: w.attempts + 1 });
const limited = failed(w, error);
this.note(
`could not read the land status of ${head12(w.sha)} (stored withdrawal of ` +
`${w.batchId}, ${w.attempts + 1} failed attempts): ${error.message}`
);
if (limited) return this.rateLimited(kept, stored.slice(i + 1));
continue;
}
if (newest?.state !== "success") {
@@ -1912,14 +1949,32 @@ class Tick {
);
this.retired.push(w.sha);
} catch (error) {
kept.push({ ...w, attempts: w.attempts + 1 });
const limited = failed(w, error);
this.note(
`${w.batchId}: withdrawal of ${statusContext} on ${head12(w.sha)} refused again ` +
`(${w.attempts + 1} failed attempts): ${String(error?.message ?? error).slice(0, 200)}`
);
if (limited) return this.rateLimited(kept, stored.slice(i + 1));
}
}
return kept;
return { kept, stopped: false };
}
/** retryList stopped by a rate limit: what it kept, then the untouched rest of the list. */
rateLimited(kept, rest) {
this.note("stored withdrawals: rate limited, the rest waits for the next tick");
return { kept: [...kept, ...rest], stopped: true };
}
/** One `error` event when a stored withdrawal's attempts reach WITHDRAWAL_ALERT_ATTEMPTS. */
alertAttempts(entry, field) {
const before = this.s.history.length;
this.s = withAttemptsAlert(this.s, entry, field, {
at: this.now(),
url: this.deps.runUrl ?? undefined,
statusContext: this.config.landing.statusContext,
});
if (this.s.history.length !== before) this.note(this.s.history.at(-1).detail);
}
async afterLanded(f, sha) {
@@ -2542,7 +2597,8 @@ class Tick {
const startedAt = pushed === "existed" ? this.retryCutoff() : pushedAt;
const rest = { ...f };
delete rest.timedOutRuns; // carried only while every run so far timed out
delete rest.landAttempts; // counted only at the landing stage, which a retry leaves
// landAttempts stays: the same candidate SHA is tested again (only a new candidate or an
// operator's pause or resume starts the count again).
delete rest.rerun; // a new ref: its runs are not the re-run's (the retry is never re-run)
delete rest.freezeHold; // a new run, a new startedAt: the in-flight clock starts again
this.s.inFlight = {
@@ -2552,6 +2608,7 @@ class Tick {
attempts,
startedAt,
stage: "testing",
...(storedLandAttempts(f) > 0 ? { landAttempts: storedLandAttempts(f) } : {}),
...(everyRunTimedOut ? { timedOutRuns: attempts } : {}),
};
this.s.counters.ciRuns += 1;
@@ -3572,9 +3629,9 @@ function withWithdrawalList(state, field, list) {
* state.landWithdrawalsOverflow (same shape, same limit). Nothing is ever dropped to make room.
* @param {QueueState} state
* @param {{sha: string, batchId: string, candidateSha: string, why: string, at: string}} refusal
* @returns {{state: QueueState, list: string|null}} the list that holds it now, or null when both
* are full (the state is then unchanged and the caller names the commit,
* recordFullWithdrawals)
* @returns {{state: QueueState, list: string|null, added: boolean, entry: object|null}} the list
* that holds it now (null when both are full: the state is then unchanged and the caller names
* the commit, recordFullWithdrawals), whether that is a new entry, and the entry
*/
export function storeWithdrawalIn(state, refusal) {
for (const field of WITHDRAWAL_LISTS) {
@@ -3582,25 +3639,75 @@ export function storeWithdrawalIn(state, refusal) {
const i = list.findIndex((w) => w.sha === refusal.sha);
if (i < 0) continue;
list[i] = { ...list[i], why: refusal.why, attempts: list[i].attempts + 1 };
return { state: withWithdrawalList(state, field, list), list: field };
return {
state: withWithdrawalList(state, field, list),
list: field,
added: false,
entry: list[i],
};
}
const { sha, batchId, candidateSha, why, at } = refusal;
for (const field of WITHDRAWAL_LISTS) {
const list = state[field] ?? [];
if (list.length >= LAND_WITHDRAWALS_LIMIT) continue;
const entry = { sha, batchId, candidateSha, why, since: at, attempts: 1 };
return { state: withWithdrawalList(state, field, [...list, entry]), list: field };
return {
state: withWithdrawalList(state, field, [...list, entry]),
list: field,
added: true,
entry,
};
}
return { state, list: null };
return { state, list: null, added: false, entry: null };
}
/**
* Whether storing a refusal (storeWithdrawalIn's result) is the full-list condition: a NEW entry
* in the overflow, or none at all (both lists full). A refusal of a SHA the overflow already holds
* is not: the main list may have room again by then.
*/
function isFullListRefusal({ list, added }) {
return list === null || (list === "landWithdrawalsOverflow" && added);
}
/**
* Failed attempts of one stored withdrawal after which one `error` event says it is still not
* withdrawn (about a day of five-minute ticks). The queue keeps retrying: an entry never expires.
*/
export const WITHDRAWAL_ALERT_ATTEMPTS = 288;
/**
* `state` plus the `error` event of a stored withdrawal whose attempts just reached
* WITHDRAWAL_ALERT_ATTEMPTS (pure); `state` unchanged at any other count.
*/
function withAttemptsAlert(state, entry, field, { at, url, statusContext }) {
if (entry?.attempts !== WITHDRAWAL_ALERT_ATTEMPTS) return state;
const detail =
`${entry.batchId}: the withdrawal of ${statusContext} on ${entry.sha} (${field}) has ` +
`failed ${WITHDRAWAL_ALERT_ATTEMPTS} times since ${entry.since}; the queue keeps retrying it`;
return appendEvent(
state,
stripUndefined({ at, type: "error", batchId: entry.batchId, sha: entry.sha, detail, url })
);
}
/** Start of the pause detail and of the `error` event of a full landWithdrawals list. */
export const WITHDRAWALS_FULL_DETAIL = "landWithdrawals is full";
/** Words that introduce, in paused.detail, the commits a full pair of lists could not keep. */
export const KEPT_NOWHERE = "Kept nowhere (both lists full)";
/**
* Landings of `f` that ended without a verdict, as stored: `landAttempts` at any stage (the
* failure path keeps it when it sends a batch back to "testing"); a batch at the landing stage of
* a document written before the count counts as 1.
* Pause reasons a full withdrawal list leaves as they are: none of them lifts by itself and
* nothing is blessed under one. A `base-red` pause does both (the base-repair lane lands under it,
* maybeAutoResume lifts it when the tip moves), so the full-list pause replaces it.
*/
const KEPT_PAUSE_REASONS = Object.freeze(["manual", "error", "infra", "tree-divergence"]);
/**
* Landings of `f` that ended without a verdict, as stored: `landAttempts` at any stage (the count
* belongs to the candidate, not to the stage); a batch at the landing stage of a document written
* before the count counts as 1.
* @param {InFlight} f
*/
function storedLandAttempts(f) {
@@ -3610,31 +3717,49 @@ function storedLandAttempts(f) {
/**
* `state` with its in-flight batch at stage "testing" when it was at "landing" (pure); every
* other field stays. `keepCount` (main()'s failure path) keeps `landAttempts`, so the attempts
* limit still counts the landings that ended without a verdict; a pause resets it, as
* Tick.leaveLandingStage does.
* other field stays, its landing count included (written out: a document of the code before the
* count has none at the landing stage, where it counts as 1). Only a new candidate and an
* operator's pause or resume start the count again.
*/
function leaveLandingIn(state, { keepCount = false } = {}) {
function leaveLandingIn(state) {
const f = state.inFlight;
if (f?.stage !== "landing") return state;
const rest = { ...f, stage: "testing" };
if (!keepCount) delete rest.landAttempts;
return { ...state, inFlight: rest };
return { ...state, inFlight: { ...f, stage: "testing", landAttempts: storedLandAttempts(f) } };
}
/** "<sha> (<batchId>)", comma-separated. */
const namedCommits = (list) => list.map((r) => `${r.sha} (${r.batchId})`).join(", ");
/**
* The commits of `list`, named in full, as many as fit in `room` characters, then
* "and N more, named in paused.detail".
*/
function namedWithin(list, room) {
for (let n = list.length; n > 0; n--) {
const more = n < list.length ? `, and ${list.length - n} more, named in paused.detail` : "";
const text = namedCommits(list.slice(0, n)) + more;
if (text.length <= room) return text;
}
return `${list.length} commit(s), named in paused.detail`;
}
/**
* Record (pure) the refusals of a tick that found state.landWithdrawals full: those it kept in
* state.landWithdrawalsOverflow (`list` "landWithdrawalsOverflow") and those it could keep nowhere
* (`list` null, both lists full). A full list means GitHub refuses the App's status writes, so the
* queue must not go on blessing:
* - a base that is not paused is paused with reason `error`; the detail says how many entries
* each list holds and names in full each commit kept nowhere (the only record of it, with the
* event below); the batch in flight leaves the landing stage (its count reset, as for any
* pause);
* - a base already paused keeps its pause untouched (reason, detail, since), except a pause of
* this kind, which gets the commits kept nowhere appended;
* Record (pure) the refusals of a tick that found state.landWithdrawals full: those it kept as new
* entries of state.landWithdrawalsOverflow (`list` "landWithdrawalsOverflow") and those it could
* keep nowhere (`list` null, both lists full). A full list means GitHub refuses the App's status
* writes, so the queue must not go on blessing:
* - a base that is not paused, or paused as `base-red` (which lifts by itself and blesses its
* repair lane), is paused with reason `error`: the detail says how many entries each list
* holds, names in full each commit kept nowhere and, for a replaced base-red pause, says so in
* one clause; the batch in flight leaves the landing stage (its count kept);
* - a base paused for a reason of KEPT_PAUSE_REASONS keeps its reason and `since`; a commit kept
* nowhere is appended to its detail (`Kept nowhere (both lists full): <sha> (<batchId>)`), so
* paused.detail names every such commit, whatever the pause;
* - in every case one `error` event says the same, its `sha` the first commit kept nowhere, else
* the first kept in the overflow.
* the first kept in the overflow; its detail is cut at 500 characters, so it names the commits
* kept nowhere that fit and how many more paused.detail holds.
* paused.detail has no length bound in validateState: every commit fits there (the dashboard and
* the CLI show a cut copy).
* @param {QueueState} state
* @param {{sha: string, batchId: string, list: string|null}[]} refusals
* @param {{at: string, url?: string, statusContext: string}} options
@@ -3645,44 +3770,49 @@ export function recordFullWithdrawals(state, refusals, { at, url, statusContext
const all = [...bySha.values()];
if (!all.length) return state;
const nowhere = all.filter((r) => r.list === null);
const named = (list) => list.map((r) => `${r.sha} (${r.batchId})`).join(", ");
const counts =
`${(state.landWithdrawals ?? []).length} of ${LAND_WITHDRAWALS_LIMIT}; ` +
`landWithdrawalsOverflow holds ${(state.landWithdrawalsOverflow ?? []).length} of ` +
`${LAND_WITHDRAWALS_LIMIT}`;
const lost = nowhere.length
? `. Kept nowhere (both lists full), the only record of them: ${named(nowhere)}; check ` +
`their ${statusContext} by hand (docs/ops/OMNI_QUEUE.md, Incidents)`
: "";
const paused = state.paused;
const ours =
paused?.reason === "error" && String(paused.detail ?? "").startsWith(WITHDRAWALS_FULL_DETAIL);
const kept = paused && KEPT_PAUSE_REASONS.includes(paused.reason);
let next = state;
if (!paused) {
if (!kept) {
const lost = nowhere.length
? `. ${KEPT_NOWHERE}: ${namedCommits(nowhere)}; check their ${statusContext} by hand ` +
"(docs/ops/OMNI_QUEUE.md, Incidents)"
: "";
const replaced = paused
? `; the base was also paused as ${paused.reason}: ${String(paused.detail).slice(0, 200)}`
: "";
const detail =
`${WITHDRAWALS_FULL_DETAIL} (${counts}): GitHub refused the App's ${statusContext} ` +
"failure again; every refused withdrawal is kept and retried on every tick. Find out why " +
"GitHub refuses the App's status writes, wait until both lists are empty, then resume" +
lost;
lost +
replaced;
next = leaveLandingIn({
...state,
paused: { reason: "error", since: at, baseSha: null, detail, issue: null },
});
} else if (ours) {
const fresh = nowhere.filter((r) => !paused.detail.includes(r.sha));
} else {
const fresh = nowhere.filter((r) => !String(paused.detail).includes(r.sha));
if (fresh.length) {
const detail = `${paused.detail}; also kept nowhere: ${named(fresh)}`;
const detail = `${paused.detail}; ${KEPT_NOWHERE}: ${namedCommits(fresh)}`;
next = { ...state, paused: { ...paused, detail } };
}
}
const overflowed = all.filter((r) => r.list !== null).map((r) => head12(r.sha));
const detail =
const head =
`${WITHDRAWALS_FULL_DETAIL} (${counts})` +
(overflowed.length
? `: refused withdrawals kept in the overflow: ${overflowed.join(", ")}`
: "") +
(nowhere.length ? `; kept nowhere: ${named(nowhere)}` : "") +
(paused && !ours ? `; the base stays paused (${paused.reason})` : "");
(kept ? `; the base stays paused (${paused.reason})` : "") +
(paused && !kept ? `; the ${paused.reason} pause was replaced` : "");
const detail = nowhere.length
? `${head}; kept nowhere: ${namedWithin(nowhere, 500 - head.length - 16)}`
: head;
const first = nowhere[0] ?? all[0];
return appendEvent(
next,
@@ -3701,8 +3831,8 @@ export function recordFullWithdrawals(state, refusals, { at, url, statusContext
* Apply to the stored state what a tick that threw left for it (failedTickRecord), pure: drop
* the stored withdrawals it settled, take the batch it withdrew off the landing stage when the
* stored batch is still that one at that stage (its landing count kept), store the withdrawals
* GitHub refused (and record a full list, recordFullWithdrawals). A null record
* returns `state` unchanged.
* GitHub refused (with the alert of an entry that reaches WITHDRAWAL_ALERT_ATTEMPTS, and record a
* full list, recordFullWithdrawals). A null record returns `state` unchanged.
* @param {QueueState} state
* @param {ReturnType<typeof failedTickRecord>} record
* @param {{at: string, url?: string, statusContext: string}} options
@@ -3728,13 +3858,13 @@ export function amendFailedTick(state, record, options) {
f.batchId === leave.batchId &&
f.candidateSha === leave.candidateSha
) {
next = leaveLandingIn(next, { keepCount: true });
next = leaveLandingIn(next);
}
const full = [];
for (const refusal of record.withdrawals ?? []) {
const result = storeWithdrawalIn(next, refusal);
next = result.state;
if (result.list !== "landWithdrawals") full.push({ ...refusal, list: result.list });
next = withAttemptsAlert(result.state, result.entry, result.list, options);
if (isFullListRefusal(result)) full.push({ ...refusal, list: result.list });
}
return full.length ? recordFullWithdrawals(next, full, options) : next;
}

View File

@@ -29,11 +29,13 @@ import { NO_HEARTBEAT_LINE, assessHealth } from "../../../scripts/merge-queue/da
import {
GH_WRITE_METHODS,
LANDING_ATTEMPTS_LIMIT,
WITHDRAWAL_ALERT_ATTEMPTS,
RERUN_START_MINUTES,
STATE_HEARTBEAT_MINUTES,
BlessRefusedError,
StateConflictError,
TICK_ACTIONS,
amendFailedTick,
createTickDeps,
discoverBases,
failedTickRecord,
@@ -41,6 +43,7 @@ import {
main,
parseArgs,
readOnlyGitHub,
recordFullWithdrawals,
recordTickFailure,
runTick,
WITHDRAWALS_FULL_DETAIL,
@@ -1573,7 +1576,7 @@ test("a red batch at the landing stage leaves it before its failed jobs are re-r
const state = await h.state();
assert.equal(state.history.at(-1).type, "flaky-rerun");
assert.equal(state.inFlight.stage, "testing");
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
assert.equal(state.inFlight.rerun.ref, f.candidateRef);
assert.equal(newestLand(h, f.candidateSha), "failure", "its land status is withdrawn");
assert.deepEqual(h.world.lands, []);
@@ -2015,7 +2018,7 @@ test("a red batch at the landing stage loses its land status on the first red ti
assert.deepEqual(types(state), ["batch-started"], "it only waits");
assert.deepEqual(rerunCalls(h), []);
assert.equal(state.inFlight.stage, "testing");
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
assert.equal("rerun" in state.inFlight, false);
assert.equal(newestLand(h, f.candidateSha), "failure", "its land status is withdrawn");
assert.deepEqual(h.world.lands, []);
@@ -3142,7 +3145,7 @@ test("a landing-stage batch whose checks went back to pending is withdrawn and r
let state = await h.state();
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
assert.equal(state.inFlight.stage, "testing");
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
assert.deepEqual(h.world.lands, []);
assert.equal(newestLand(h, f.candidateSha), "failure", "not blessed while its checks run");
assert.deepEqual(statusStates(h), ["failure"]);
@@ -3160,8 +3163,8 @@ test("a landing-stage batch whose checks went back to pending is withdrawn and r
.filter((message) => message.startsWith(`omni-queue: landing ${f.batchId} `));
assert.deepEqual(
marker,
[`omni-queue: landing ${f.batchId} on ${BASE}`],
"the landing stage is stored again before the blessing, as attempt 1: a hold resets the count"
[`omni-queue: landing ${f.batchId} on ${BASE} (attempt 2)`],
"the landing stage is stored again before the blessing, as attempt 2: the count is the candidate's (W3)"
);
});
@@ -3607,7 +3610,7 @@ test("live → shadow on a paused base: a batch left at the landing stage loses
assert.equal(state.paused?.reason, "infra", "still paused");
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
assert.equal(state.inFlight.stage, "testing", "shadow keeps no batch at the landing stage");
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
assert.equal(newestLand(h, f.candidateSha), "failure");
});
@@ -4001,9 +4004,9 @@ for (const blessed of [false, true]) {
assert.match(held[0].detail, /^base-frozen: freeze on release\/v9\.9\.9/);
assert.equal(state.history.filter((e) => e.type === "checks-green").length, 1);
assert.deepEqual(h.writesOf("removeLabel"), []);
// Held = not at the landing stage: a blessing is withdrawn, and the landing count goes too.
// Held = not at the landing stage: a blessing is withdrawn; the landing count stays (W3).
assert.equal(state.inFlight.stage, "testing");
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, blessed ? 1 : undefined, "the candidate's count");
const withdrawn = blessed ? [landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })] : [];
assert.deepEqual(
statusCalls(h).map((c) => c.args),
@@ -7223,7 +7226,7 @@ test("M1 probe: a landing-stage batch whose base moved and whose recomposition p
assert.equal(newestLand(h, f.candidateSha), "failure");
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
assert.equal(state.inFlight.stage, "testing", `tick ${tick}: no landing claimed`);
assert.equal("landAttempts" in state.inFlight, false);
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
assert.equal(state.failedTicks, tick);
assert.equal(
state.history.findLast((e) => e.type === "error").detail.includes("hung up"),
@@ -7729,7 +7732,7 @@ test("V1 (review probe 1): landings that throw, alternating with ticks that thro
assert.equal(newestLand(h, f.candidateSha), "failure");
});
test("V1: a freeze hold between two failing landings still resets the count", async () => {
test("W3 (was V1): a freeze hold between two failing landings keeps the count", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
await greenBatch(h, [1]);
const getPull = h.gh.getPull;
@@ -7744,12 +7747,12 @@ test("V1: a freeze hold between two failing landings still resets the count", as
await h.tick();
let state = await h.state();
assert.equal(state.inFlight.stage, "testing");
assert.equal("landAttempts" in state.inFlight, false, "the hold resets it");
assert.equal(state.inFlight.landAttempts, 1, "the hold keeps it");
freeze.state = "closed";
h.clock.advance(10);
await recordedFailingTick(h, /not available from origin/);
state = await h.state();
assert.equal(state.inFlight.landAttempts, 1, "counted from 1 again");
assert.equal(state.inFlight.landAttempts, 2, "counted on");
});
test("V4: a batch loaded at the landing stage and recomposed has its status withdrawn before the queue snapshot", async () => {
@@ -7872,7 +7875,8 @@ test("V5: the failure path leaves a batch whose candidate is the base tip at the
const getBranchSha = h.gh.getBranchSha;
let reads = 0;
h.gh.getBranchSha = async (ref) => {
if (ref === BASE && reads++ === 0) throw new Error("GET /git/ref → 502: Bad Gateway");
// both reads of recordEarlierLanding fail (W4); the failure path's own read answers
if (ref === BASE && reads++ < 2) throw new Error("GET /git/ref → 502: Bad Gateway");
return getBranchSha(ref);
};
const listCheckRuns = h.gh.listCheckRuns;
@@ -7889,3 +7893,288 @@ test("V5: the failure path leaves a batch whose candidate is the base tip at the
assert.equal(state.history.filter((e) => e.type === "landed").length, 1);
assert.deepEqual(statusCalls(h), []);
});
// ─────────── Task 13b fix round 2 (W1 to W4) ───────────
/** 20 + 20 stored withdrawals of App-blessed commits: both lists full. */
async function bothListsFull(h) {
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
const spill = Array.from({ length: 20 }, (_, i) => fakeSha(`spill-${i}`));
for (const sha of [...old, ...spill]) setLand(h, sha, "success");
const { state, version } = await h.store.load(BASE);
state.landWithdrawals = old.map((sha) => storedWithdrawal(sha));
state.landWithdrawalsOverflow = spill.map((sha) => storedWithdrawal(sha));
assert.equal((await h.store.save(BASE, state, version, "seed: both lists full")).ok, true);
return { old, spill };
}
test("W1 (re-review RR-8): a base paused by hand, both lists full: the commit kept nowhere is named in that pause's detail, its reason and since untouched", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
const f = await greenBatch(h, [1]);
await bothListsFull(h);
await diedAfterBlessing(h, f, [f.candidateSha]);
refuseWithdrawals(h, () => true);
const at = h.clock.now();
await h.tick({ action: "pause", reason: "before the mode change" });
const state = await h.state();
assert.equal(state.paused.reason, "manual");
assert.equal(state.paused.since, at, "the operator's pause time");
assert.ok(state.paused.detail.startsWith("before the mode change"), state.paused.detail);
assert.ok(
state.paused.detail.includes(
`Kept nowhere (both lists full): ${f.candidateSha} (${f.batchId})`
),
state.paused.detail
);
const event = state.history.findLast((e) => e.type === "error");
assert.equal(event.sha, f.candidateSha);
});
test("W1: more commits kept nowhere than an event detail holds: all in paused.detail, the event names what fits and how many more", () => {
const heads = Array.from({ length: 12 }, (_, i) => ({
sha: fakeSha(`head-nowhere-${i}`),
batchId: "b12",
list: null,
}));
const paused = {
reason: "manual",
since: "2026-11-13T11:00:00.000Z",
baseSha: null,
detail: "op",
issue: null,
};
const next = recordFullWithdrawals({ ...initialState(BASE), paused }, heads, {
at: "2026-11-13T12:00:00.000Z",
statusContext: "omni-queue/land",
});
assert.equal(next.paused.reason, "manual");
assert.equal(next.paused.since, paused.since);
for (const head of heads) assert.ok(next.paused.detail.includes(head.sha), head.sha);
const event = next.history.at(-1);
assert.ok(event.detail.length <= 500);
const named = heads.filter((head) => event.detail.includes(head.sha)).length;
assert.ok(named >= 1 && named < 12, `named ${named}`);
assert.ok(event.detail.includes(`and ${12 - named} more, named in paused.detail`), event.detail);
});
test("W2: a base-red pause with a repair in flight is replaced by the full-list pause: nothing is blessed under it and it does not lift when the tip moves", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
const { baseRepairLabel } = h.config.lanes;
await baseRedPause(h, [1, 2]);
h.addPr(3);
h.addLane(3, baseRepairLabel);
h.world.repairs.add(3);
await h.tick();
const repair = (await h.state()).inFlight;
assert.equal(repair.lane, "base-repair");
h.runCi(repair.candidateSha);
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
for (const sha of old) setLand(h, sha, "success");
await storeWithdrawals(
h,
old.map((sha) => storedWithdrawal(sha))
);
h.world.baseMovesOnLand = fakeSha("manual-merge"); // the repair's landing is refused
refuseWithdrawals(h, () => true);
await h.tick();
let state = await h.state();
assert.equal(state.paused.reason, "error", "the base-red pause is replaced");
assert.ok(state.paused.detail.startsWith(WITHDRAWALS_FULL_DETAIL), state.paused.detail);
assert.match(state.paused.detail, /also paused as base-red/);
const blessings = statusStates(h).filter((s) => s === "success").length;
const next = state.inFlight;
if (next) h.runCi(next.candidateSha);
h.gh.branches.set(BASE, fakeSha("tip-moved-again"));
for (let i = 0; i < 2; i++) {
h.clock.advance(10);
await h.tick();
}
state = await h.state();
assert.equal(state.paused?.reason, "error", "the tip moved: still paused");
assert.equal(statusStates(h).filter((s) => s === "success").length, blessings, "nothing blessed");
});
test("W3 (re-review RR-2): landings that throw, alternating with checks back to pending, pause at the third landing without a verdict", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
const f = await greenBatch(h, [1]);
h.world.landError = notAvailable(1);
for (let attempt = 1; attempt <= 3; attempt++) {
await recordedFailingTick(h, /not available from origin/);
assert.equal((await h.state()).inFlight.landAttempts, attempt);
rerunGate(h, f.candidateSha, "in_progress");
await h.tick();
const state = await h.state();
assert.equal(state.inFlight.stage, "testing", "pending: back to testing");
assert.equal(state.inFlight.landAttempts, attempt, "the count is the candidate's: kept");
rerunGate(h, f.candidateSha, "completed");
}
await h.tick();
const state = await h.state();
assert.equal(state.paused?.reason, "error");
assert.match(state.paused.detail, /ended without a verdict 3 times in a row/);
assert.equal(statusStates(h).filter((s) => s === "success").length, 3, "three blessings");
});
test("W3 (re-review RR-10): a shadow tick never pauses on a kept count", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
await greenBatch(h, [1]);
const getPull = h.gh.getPull;
h.world.landError = notAvailable(1);
for (let i = 0; i < 3; i++) {
h.gh.getPull = getPull;
await recordedFailingTick(h, /not available from origin/);
}
h.gh.getPull = brokenGetPull();
await recordedFailingTick(h, /502/);
h.gh.getPull = getPull;
assert.equal((await h.state()).inFlight.landAttempts, 3);
h.config.mode = "shadow";
await h.tick();
const state = await h.state();
assert.equal(state.paused, null, "shadow blesses nothing, so it never pauses for the limit");
assert.ok(types(state).includes("would-land"));
});
test("W3: a recomposition starts the count again", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
const f = await greenBatch(h, [1]);
h.world.landError = notAvailable(1);
await recordedFailingTick(h, /not available from origin/);
h.world.landError = null;
h.gh.branches.set(BASE, fakeSha("manual-merge"));
await h.tick();
const state = await h.state();
assert.notEqual(state.inFlight.candidateSha, f.candidateSha, "recomposed");
assert.equal("landAttempts" in state.inFlight, false);
});
test("W3: an operator's pause, and an operator's resume, start the count again", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
await greenBatch(h, [1]);
h.world.landError = notAvailable(1);
await recordedFailingTick(h, /not available from origin/);
await h.tick({ action: "pause", reason: "incident" });
assert.equal("landAttempts" in (await h.state()).inFlight, false, "the pause resets it");
// A resume of a pause the operator did not make (here: an infra pause) resets it too.
const { state: stored, version } = await h.store.load(BASE);
stored.paused = {
reason: "infra",
since: h.clock.now(),
baseSha: null,
detail: "x",
issue: null,
};
stored.inFlight = { ...stored.inFlight, landAttempts: 2 };
assert.equal((await h.store.save(BASE, stored, version, "seed")).ok, true);
h.clock.advance(10);
h.world.landError = notAvailable(2);
await recordedFailingTick(h, /not available from origin/, { action: "resume" });
assert.equal((await h.state()).inFlight.landAttempts, 1, "the resume's own landing is attempt 1");
});
test("W4 (re-review RR-9a): a refusal of a SHA already in the overflow, with room in the main list, is not a full list", () => {
const sha = (n) => n.toString(16).padStart(40, "0");
const entry = (n) => storedWithdrawal(sha(n), { candidateSha: sha(n) });
const state = {
...initialState(BASE),
landWithdrawals: [entry(1), entry(2)],
landWithdrawalsOverflow: [entry(3)],
};
const refusal = {
sha: sha(3),
batchId: "b9",
candidateSha: sha(3),
why: "y",
at: "2026-11-13T12:00:00.000Z",
};
const next = amendFailedTick(
state,
{ leaveLanding: null, withdrawals: [refusal], retired: [] },
{ at: "2026-11-13T12:00:00.000Z", statusContext: "omni-queue/land" }
);
assert.equal(next.paused, null);
assert.equal(next.landWithdrawalsOverflow[0].attempts, 2);
assert.ok(!next.history.some((e) => e.type === "error"));
});
test("W4: stored retries stop at a rate limit: the rest of both lists waits for the next tick", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
await h.tick({ action: "pause", reason: "incident" });
const [ok, limited, later, spilled] = ["ok", "limited", "later", "spilled"].map(fakeSha);
for (const sha of [ok, limited, later, spilled]) setLand(h, sha, "success");
const { state: stored, version } = await h.store.load(BASE);
stored.landWithdrawals = [ok, limited, later].map((sha) => storedWithdrawal(sha));
stored.landWithdrawalsOverflow = [storedWithdrawal(spilled)];
assert.equal((await h.store.save(BASE, stored, version, "seed")).ok, true);
const list = h.gh.listCommitStatuses;
h.gh.listCommitStatuses = async (sha) => {
if (sha === limited) {
throw Object.assign(
new Error("GET statuses → 403: API rate limit exceeded for installation"),
{
name: "GitHubError",
status: 403,
}
);
}
return list(sha);
};
h.clock.advance(10);
await h.tick();
const state = await h.state();
assert.equal(landStatusReads(h, later), 0, "not read after the rate limit");
assert.equal(landStatusReads(h, spilled), 0, "the overflow neither");
assert.equal(newestLand(h, ok), "failure", "the one before it was written");
assert.deepEqual(
state.landWithdrawals.map((w) => [w.sha, w.attempts]),
[
[limited, 2],
[later, 1],
]
);
assert.deepEqual(
state.landWithdrawalsOverflow.map((w) => [w.sha, w.attempts]),
[[spilled, 1]]
);
});
test("W4: one error event when a stored withdrawal reaches WITHDRAWAL_ALERT_ATTEMPTS, not on the ticks after", async () => {
assert.equal(WITHDRAWAL_ALERT_ATTEMPTS, 288);
const h = createHarness({ mode: "live", actor: "owner" });
await h.tick({ action: "pause", reason: "incident" });
const sha = fakeSha("stubborn");
setLand(h, sha, "success");
await storeWithdrawals(h, [storedWithdrawal(sha, { attempts: 287 })]);
refuseWithdrawals(h, () => true);
const alerts = (state) =>
state.history.filter((e) => e.type === "error" && e.sha === sha && /288/.test(e.detail ?? ""));
h.clock.advance(10);
await h.tick();
let state = await h.state();
assert.equal(state.landWithdrawals[0].attempts, 288);
assert.equal(alerts(state).length, 1);
assert.match(alerts(state)[0].detail, /landWithdrawals/);
assert.match(alerts(state)[0].detail, /keeps retrying/);
h.clock.advance(10);
await h.tick();
state = await h.state();
assert.equal(state.landWithdrawals[0].attempts, 289);
assert.equal(alerts(state).length, 1, "once, at the crossing");
});
test("W4: the tip read of recordEarlierLanding is tried twice before falling back", async () => {
const h = createHarness({ mode: "live", actor: "owner" });
const f = await greenBatch(h, [1]);
await landedButNotSaved(h, f);
const getBranchSha = h.gh.getBranchSha;
let reads = 0;
h.gh.getBranchSha = async (ref) => {
if (ref === BASE && reads++ === 0) throw new Error("GET /git/ref → 502: Bad Gateway");
return getBranchSha(ref);
};
await h.tick({ action: "pause", reason: "incident" });
const state = await h.state();
assert.deepEqual(statusCalls(h), [], "no failure on the landed tip");
assert.equal(state.history.filter((e) => e.type === "landed").length, 1);
});