mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-11-18 07:58:27 +00:00
fix(ci): omni-queue landing count belongs to the candidate; full lists never hide a commit
Task 13b, fix round 2, rulings W1, W2, W3, W4 (code items) and W5.
W1: when both lists are full and the base is already paused for a kept
reason, "Kept nowhere (both lists full): <sha> (<batchId>)" is appended to
that pause's detail (reason and since untouched), so paused.detail names
every commit kept nowhere. validateState has no bound on paused.detail;
the error event's detail is cut at 500 characters, so it names the
commits that fit and "and N more, named in paused.detail" (namedWithin).
W2: only manual, error, infra and tree-divergence pauses are kept
(KEPT_PAUSE_REASONS: none lifts by itself, nothing is blessed under one).
A base-red pause is replaced by the full-list pause, whose detail says
the base was also paused as base-red (the repair lane blesses under it,
maybeAutoResume lifts it when the tip moves).
W3: inFlight.landAttempts belongs to the candidate. Checks back to
pending, a re-run, a release-freeze hold, a paused base leaving it as is,
an infra retry (same SHA), the failure path and the full-list pause keep
it (written out: a stored landing without the field counts as 1); only a
new candidate and an operator's pause or resume (handleAction) start it
again. The limit is checked in live mode only: a shadow tick never pauses
on a carried count. Contract comment, tick header and runbook updated.
W4: only a NEW entry of the overflow is the full-list condition
(isFullListRefusal; storeWithdrawalIn says whether the entry is new). A
read or a write of a stored withdrawal that fails with a rate limit
(isRateLimitError) stops the retries of the tick; the rest of both lists
waits. One error event when an entry's attempts reach
WITHDRAWAL_ALERT_ATTEMPTS (288), at the crossing only, from the retry,
the in-tick store and the failure path. recordEarlierLanding reads the
base tip twice before falling back.
W5 runbook: the withdrawal is written once, by the pass right after the
batch in flight; the attempts sentence; the full-list incident (a NEW
overflow entry; which pauses are kept; base-red replaced; the counts on
the dashboard and in status); where a commit kept nowhere is named and
how long each view is; step 2 of a mode change reads paused.detail and
the history with --limit 200; the last-resort hand edit says what
disabling the queue-refs ruleset does to other live bases and that the
kill switch avoids it; an unknown commit's statuses answer an empty list.
New tests (tick.test.mjs), failed on 021ff70 unless marked:
- "W1 (re-review RR-8): a base paused by hand, both lists full: the
commit kept nowhere is named in that pause's detail, its reason and
since untouched"
- "W1: more commits kept nowhere than an event detail holds: all in
paused.detail, the event names what fits and how many more"
- "W2: a base-red pause with a repair in flight is replaced by the
full-list pause: nothing is blessed under it and it does not lift when
the tip moves"
- "W3 (re-review RR-2): landings that throw, alternating with checks back
to pending, pause at the third landing without a verdict"
- "W3 (re-review RR-10): a shadow tick never pauses on a kept count"
- "W3: a recomposition starts the count again" (passed at once: pins it)
- "W3: an operator's pause, and an operator's resume, start the count
again"
- "W4 (re-review RR-9a): a refusal of a SHA already in the overflow, with
room in the main list, is not a full list"
- "W4: stored retries stop at a rate limit: the rest of both lists waits
for the next tick"
- "W4: one error event when a stored withdrawal reaches
WITHDRAWAL_ALERT_ATTEMPTS, not on the ticks after"
- "W4: the tip read of recordEarlierLanding is tried twice before falling
back"
Re-aimed to W3 (each asserted that landAttempts was gone; each now
asserts the candidate's count is kept):
- "a red batch at the landing stage leaves it before its failed jobs are
re-run"
- "a red batch at the landing stage loses its land status on the first red
tick, while it waits"
- "a landing-stage batch whose checks went back to pending is withdrawn
and re-blessed when green" (its landing marker is now "(attempt 2)")
- "live → shadow on a paused base: a batch left at the landing stage loses
its status too"
- "a batch at the landing stage held by a release-freeze: its land status
turns failure; blessed again when the freeze lifts and it lands" (1
when blessed, none otherwise)
- "M1 probe: a landing-stage batch whose base moved and whose
recomposition push throws six times: withdrawn by the first tick, not
stored as landing after it" (count 1 kept)
- round 1's "V1: a freeze hold between two failing landings still resets
the count" is now "W3 (was V1): a freeze hold between two failing
landings keeps the count" (kept through the hold, then 2)
Re-aimed to W4: "V5: the failure path leaves a batch whose candidate is
the base tip at the landing stage, with no write; the next tick records
it" now makes both of recordEarlierLanding's reads fail (it read once).
This commit is contained in:
@@ -190,27 +190,32 @@ shadow tick that meets such a batch records it the same way; shadow mode lands n
|
||||
records a landing a live tick made.
|
||||
|
||||
A withdrawal GitHub refuses while the landing withdraws its own blessing (the landing ended with a
|
||||
verdict other than landed) is tried once more by the end-of-tick pass above; other refusals wait
|
||||
for the next tick. Every refused write is recorded with an `error` event, detail
|
||||
verdict other than landed) is tried once more by the pass that runs right after the batch in
|
||||
flight is handled (below); other refusals wait for the next tick. Every refused write is recorded with an `error` event, detail
|
||||
`<batch>: could not withdraw <landing.statusContext> (<why>) on <sha12> (<error>)`, and stored in
|
||||
the state field `landWithdrawals` (see [State](#state)). Every later tick, on a paused base too,
|
||||
retries the stored withdrawals before it handles the batch in flight: it reads the land statuses
|
||||
of each stored commit, writes `failure` where the same rule finds a `success` (the App's newest,
|
||||
or the newest of the App's and those without a readable creator), and drops the entry once that
|
||||
write succeeds or neither is `success` any more (or there is no status). A refused retry keeps
|
||||
the entry and counts an attempt, and so does a failed read: `attempts` is the number of failed
|
||||
attempts so far (a read that failed or a write that was refused). No count drops an entry: the
|
||||
queue never gives up on withdrawing a blessing. A commit the tick blesses for landing, or an
|
||||
approver blesses by hand, leaves the list.
|
||||
write succeeds or neither is `success` any more (or there is no status: GitHub answers an empty
|
||||
list for a commit it does not know, so such an entry leaves at its next retry). A refused retry
|
||||
keeps the entry and counts an attempt, and so does a failed read: `attempts` is the number of
|
||||
failed attempts so far (a read that failed or a write that was refused). No count drops an entry:
|
||||
the queue never gives up on withdrawing a blessing. When an entry's `attempts` reach 288
|
||||
(`WITHDRAWAL_ALERT_ATTEMPTS` in `scripts/merge-queue/tick.mjs`, about a day of ticks), one `error`
|
||||
event names the commit and its list and says the queue keeps retrying. A read or a write that
|
||||
fails with a rate limit stops the retries of that tick: the rest of both lists waits for the next
|
||||
tick. A commit the tick blesses for landing, or an approver blesses by hand, leaves the list.
|
||||
Shadow mode retries the stored withdrawals too, and withdraws the status of a batch stored at the
|
||||
landing stage, so a switch from live to shadow strands no blessing (see
|
||||
[Before a mode change or a stop](#before-a-mode-change-or-a-stop)); a queue that was never live
|
||||
has nothing to withdraw.
|
||||
|
||||
A batch the tick loaded at the landing stage and no longer keeps there loses its blessing twice
|
||||
over: right after the batch in flight is handled, before the queue snapshot and any composition
|
||||
(which could hang until the job's timeout), and again at the end of the tick for what the rest of
|
||||
the tick changed (the same read, once per batch).
|
||||
A batch the tick loaded at the landing stage and no longer keeps there loses its blessing once,
|
||||
right after the batch in flight is handled, before the queue snapshot and any composition (which
|
||||
could hang until the job's timeout). The end of the tick runs the same pass again, which writes
|
||||
only for a batch that left the landing stage later in the tick (a pause that a full withdrawal
|
||||
list caused, for example).
|
||||
|
||||
A tick that throws (any error but a lost compare-and-swap; never in a dry-run) saves no state of
|
||||
its own after the error; what it saved before (the landing stage and its count, written before a
|
||||
@@ -235,11 +240,18 @@ The list holds at most 20 entries (`LAND_WITHDRAWALS_LIMIT` in
|
||||
is stored in the state field `landWithdrawalsOverflow` (same entries, at most 20 more), and both
|
||||
lists are retried by the same code on every tick, in every mode. A full main list means GitHub is
|
||||
refusing the App's status writes, so a base that is not paused yet is paused with reason `error`; the
|
||||
detail starts with `landWithdrawals is full` and says how many entries each list holds. A base
|
||||
already paused keeps its own pause (reason, detail, time); an `error` event records the full list.
|
||||
Only a refusal that fits in neither list is kept nowhere: its commit is named in full in the pause
|
||||
detail (a pause of this kind gets it appended) and in that `error` event (see
|
||||
[Incidents](#incidents)).
|
||||
detail starts with `landWithdrawals is full` and says how many entries each list holds. Only a NEW
|
||||
entry in the overflow is that condition: a refusal of a commit the overflow already holds is not.
|
||||
A base already paused for a reason that neither lifts by itself nor blesses (`manual`, `error`,
|
||||
`infra`, `tree-divergence`) keeps its pause (reason, time); an `error` event records the full
|
||||
list. A `base-red` pause is replaced by the full-list pause, whose detail says the base was also
|
||||
paused as base-red: under a base-red pause the base-repair lane is still blessed, and the pause
|
||||
lifts by itself when the base tip moves. Only a refusal that fits in neither list is kept nowhere:
|
||||
its commit is named in full, after the words `Kept nowhere (both lists full)`, in `paused.detail`
|
||||
whatever the pause (appended to a kept pause's detail), and in that `error` event, whose detail is
|
||||
cut at 500 characters: it names the commits that fit and how many more `paused.detail` holds
|
||||
(see [Incidents](#incidents)). The state has no bound on `paused.detail`; the dashboard (300
|
||||
characters) and `npm run queue -- status` (200) show it cut.
|
||||
|
||||
In live mode:
|
||||
|
||||
@@ -260,8 +272,11 @@ In live mode:
|
||||
counted in `inFlight.landAttempts` and saved before its blessing, so a tick that died after the
|
||||
blessing counts too. After `LANDING_ATTEMPTS_LIMIT` (3, in `scripts/merge-queue/tick.mjs`)
|
||||
attempts in a row without a verdict the batch is not blessed again and the base is paused with
|
||||
reason `error` (see [Incidents](#incidents)). The count goes with the landing stage: a held batch,
|
||||
a manual pause or an infra retry starts it again.
|
||||
reason `error` (see [Incidents](#incidents)). The count belongs to the candidate, not to the
|
||||
stage: checks back to pending, a re-run, a release-freeze hold, an infra retry and a failed tick
|
||||
keep it; only a new candidate (a recomposition, a new batch) and an operator's pause or resume
|
||||
start it again. Shadow mode never checks the limit (it blesses nothing), so a count carried from
|
||||
live mode never pauses a shadow tick.
|
||||
- **A per-PR merge call of `squash-api` that fails** (a rate limit included) is not retried: the
|
||||
error is caught inside `landSquashApi` and is a verdict, `merge-failed`, so the base is paused
|
||||
with reason `error`. The rate-limit rule above decides the token hint here too: with
|
||||
@@ -515,11 +530,11 @@ candidate in flight (refs, SHAs, PRs, stage, attempts, bisection), the last base
|
||||
shadow bookkeeping, counters and the last 200 history events. Three fields belong to the land status
|
||||
(see [Who lands](#who-lands-landingactor)):
|
||||
|
||||
- `inFlight.landAttempts`: the number of ticks in a row that blessed the batch without reaching a
|
||||
verdict. Present at the stage `landing`, and at the stage `testing` only when a failed tick sent
|
||||
the batch back there (the failure path keeps the count); a hold, checks back to pending, a re-run,
|
||||
a pause, an infra retry or a recomposition deletes it. At `LANDING_ATTEMPTS_LIMIT` (3) the queue
|
||||
pauses instead of blessing it again.
|
||||
- `inFlight.landAttempts`: the number of ticks that blessed the candidate without reaching a
|
||||
verdict. It belongs to the candidate, at either stage: a hold, checks back to pending, a re-run,
|
||||
an infra retry and a failed tick keep it; a new candidate (a recomposition, a new batch) and an
|
||||
operator's pause or resume start it again. At `LANDING_ATTEMPTS_LIMIT` (3) a live tick pauses
|
||||
instead of blessing it again; shadow mode never checks it.
|
||||
- `landWithdrawals`: present only when it is not empty. Each entry is a commit whose `failure` land
|
||||
status GitHub refused to write (`sha`, `batchId`, `candidateSha`, `why`, `since`, `attempts`),
|
||||
oldest first, one per SHA, at most 20. Later ticks retry them.
|
||||
@@ -780,10 +795,11 @@ queue to shadow (or shadow to live), turning the kill switch off or rolling back
|
||||
`stage <stage>` for the batch in flight) and that both `landWithdrawals` and
|
||||
`landWithdrawalsOverflow` are empty: the fields are absent from `state/<sanitized-base>.json` on
|
||||
`omni-queue-state` (an empty list leaves no field). While entries remain, wait for the next ticks
|
||||
to retry them and check again. Also read the `error` events of the history
|
||||
(`npm run queue -- history --base <base>`) for one that starts with `landWithdrawals is full` and
|
||||
says `kept nowhere`: each commit it names may still carry the App's `success` (see
|
||||
[Incidents](#incidents), "Paused: landWithdrawals is full").
|
||||
to retry them and check again. Also read `paused.detail` in that file for the words
|
||||
`Kept nowhere (both lists full)`, and the history
|
||||
(`npm run queue -- history --base <base> --limit 200`, every stored event) for `error` events
|
||||
that start with `landWithdrawals is full`: each commit named after those words may still carry
|
||||
the App's `success` (see [Incidents](#incidents), "Paused: landWithdrawals is full").
|
||||
|
||||
3. Only then make the change. After a mode change, resume each base
|
||||
(`npm run queue -- resume <base>`).
|
||||
@@ -1519,12 +1535,17 @@ refused token. Once the cause is gone, `npm run queue -- resume <base>` recompos
|
||||
|
||||
**Paused: landWithdrawals is full.** A pause detail that starts with
|
||||
`landWithdrawals is full (<n> of 20; landWithdrawalsOverflow holds <m> of 20)` means GitHub refused
|
||||
to write the App's `failure` land status while 20 refused withdrawals were already stored: GitHub
|
||||
is refusing the App's status writes, so the queue stopped blessing. Every refused withdrawal is
|
||||
kept, in `landWithdrawals` or in `landWithdrawalsOverflow`, and both lists are retried on every
|
||||
tick of the paused base. A base that was already paused (a manual pause before a mode change, for
|
||||
example) keeps its own pause: the same condition is then only an `error` event that starts with
|
||||
`landWithdrawals is full`. Check, in order:
|
||||
to write the App's `failure` land status on a commit that had to become a new entry of the
|
||||
overflow, because 20 refused withdrawals were already stored: GitHub is refusing the App's status
|
||||
writes, so the queue stopped blessing. Every refused withdrawal is kept, in `landWithdrawals` or in
|
||||
`landWithdrawalsOverflow`, and both lists are retried on every tick of the paused base. A base that
|
||||
was already paused for a reason that neither lifts by itself nor blesses (`manual`, `error`,
|
||||
`infra`, `tree-divergence`; a manual pause before a mode change, for example) keeps its own pause:
|
||||
the same condition is then an `error` event that starts with `landWithdrawals is full`, plus, for a
|
||||
commit kept nowhere, its name appended to that pause's detail. A `base-red` pause is replaced by
|
||||
this one (its detail says so), since the base-repair lane is blessed under it and it lifts by
|
||||
itself. The dashboard and `npm run queue -- status` count the entries of each list while either
|
||||
is not empty. Check, in order:
|
||||
|
||||
1. Why GitHub refuses the App's status writes: the run's log names the error of each refused
|
||||
write, and so does an `error` event (`<batchId>: could not withdraw …`, or the detail of the
|
||||
@@ -1540,14 +1561,18 @@ example) keeps its own pause: the same condition is then only an `error` event t
|
||||
|
||||
The last resort: when both lists were full too, a refused withdrawal is kept nowhere. Its commit is
|
||||
then named in full, as `<sha> (<batchId>)` after the words `Kept nowhere (both lists full)`, in
|
||||
`paused.detail` of `state/<sanitized-base>.json` and in the `sha` and detail of that `error`
|
||||
event. Read it in the state file itself: the dashboard and `npm run queue -- status` cut
|
||||
long details. That commit's App status may stay `success` and nothing retries it; a person has to
|
||||
`paused.detail` of `state/<sanitized-base>.json`, whatever the pause, and in the `sha` and detail of
|
||||
that `error` event (cut at 500 characters: it names the commits that fit and how many more
|
||||
`paused.detail` holds). Read `paused.detail` in the state file itself: the dashboard (300
|
||||
characters) and `npm run queue -- status` (200) cut long details. That commit's App status may stay `success` and nothing retries it; a person has to
|
||||
put it back in a list, once a list has room (after step 2):
|
||||
|
||||
1. The base is already paused (this pause); keep it paused.
|
||||
2. Disable the `omni-queue refs protection` ruleset (Settings → Rules → Rulesets), so an ordinary
|
||||
commit can change `omni-queue-state`.
|
||||
commit can change `omni-queue-state`. While it is disabled, any other live base whose tick
|
||||
tries to bless pauses with `queue refs are not protected (…)` (see
|
||||
[Before the land status is written](#before-the-land-status-is-written)); turning the kill switch
|
||||
off first (see [Kill switch](#kill-switch)) avoids that, and you turn it on again after step 4.
|
||||
3. In `state/<sanitized-base>.json`, append to `landWithdrawals` (or to `landWithdrawalsOverflow`;
|
||||
at most 20 entries in each, and a SHA in one list only) one entry per named commit:
|
||||
`{"sha": "<40-hex sha>", "batchId": "<batchId>", "candidateSha": "<40-hex sha>", "why": "left the landing stage", "since": "<ISO 8601 time, e.g. 2026-11-16T12:00:00.000Z>", "attempts": 1}`,
|
||||
|
||||
@@ -193,14 +193,14 @@ const SAFE_SEGMENT_RE = /^[A-Za-z0-9._-]+$/;
|
||||
* timeouts, set only while that holds for EVERY run (=
|
||||
* attempts); absent otherwise, and in documents written before
|
||||
* it existed (= first run not known to be a timeout)
|
||||
* @property {number} [landAttempts] ticks that blessed this batch for landing and reached no
|
||||
* verdict, counted (and saved) before each blessing. Set at the
|
||||
* landing stage, and kept at stage "testing" only when main()'s
|
||||
* failure path sent the batch back there (tick amendFailedTick);
|
||||
* deleted by every other exit from the landing stage (hold,
|
||||
* pending checks, re-run, pause, infra retry, recomposition).
|
||||
* Absent in documents written before it existed (= 1 at the
|
||||
* landing stage). See tick LANDING_ATTEMPTS_LIMIT
|
||||
* @property {number} [landAttempts] ticks that blessed this candidate for landing and reached
|
||||
* no verdict, counted (and saved) before each blessing. It
|
||||
* belongs to the candidate, at either stage: kept through a
|
||||
* hold, pending checks, a re-run, an infra retry and main()'s
|
||||
* failure path; a new candidate (recomposition, new batch) has
|
||||
* none, and an operator's pause or resume deletes it. Absent in
|
||||
* documents written before it existed (= 1 at the landing
|
||||
* stage). See tick LANDING_ATTEMPTS_LIMIT (live mode only)
|
||||
* @property {RerunRecord} [rerun] the failed jobs of this candidate were re-run once (tick
|
||||
* onRed); absent before that, in documents written before it
|
||||
* existed, and on an infra retry (a new ref)
|
||||
|
||||
@@ -78,8 +78,10 @@
|
||||
// any later tick that drops, recomposes or pauses that batch turns the App's `success` land
|
||||
// status on its commits into `failure` (withdrawAbandoned; only statuses created by the App
|
||||
// count, as for the release ruleset). That save also counts the attempt
|
||||
// (inFlight.landAttempts): after LANDING_ATTEMPTS_LIMIT attempts in a row without a verdict
|
||||
// the batch is not blessed again and the queue pauses with reason "error". A batch HELD at
|
||||
// (inFlight.landAttempts, the candidate's: kept at either stage, started again only by a new
|
||||
// candidate and by an operator's pause or resume): after LANDING_ATTEMPTS_LIMIT attempts
|
||||
// without a verdict a live tick does not bless the batch again and the queue pauses with
|
||||
// reason "error" (shadow never checks it). A batch HELD at
|
||||
// the landing stage (release-freeze, checks back to pending, manual pause) goes back to stage
|
||||
// "testing" (leaveLandingStage), so the same tick withdraws its status and later held ticks
|
||||
// read nothing; it is blessed again only right before it lands. A paused base that does not
|
||||
@@ -89,9 +91,10 @@
|
||||
// attempt withdrew its own blessing and keeps its count). A `failure` GitHub refuses to write
|
||||
// is kept in state.landWithdrawals and retried by the next ticks (retryWithdrawals); when that
|
||||
// list is full (LAND_WITHDRAWALS_LIMIT) nothing is dropped: the refusal goes to
|
||||
// state.landWithdrawalsOverflow, retried alike, and a base that is not paused pauses with
|
||||
// reason "error" (recordFullWithdrawals); only a refusal that fits in neither list is named,
|
||||
// in full, in the pause detail and in an `error` event.
|
||||
// state.landWithdrawalsOverflow, retried alike, and a base that is not paused (or paused as
|
||||
// base-red, which blesses its repair lane and lifts by itself) pauses with reason "error"
|
||||
// (recordFullWithdrawals); only a refusal that fits in neither list is named, in full, in
|
||||
// paused.detail whatever the pause, and in an `error` event.
|
||||
// Before landing (or composing a bisection segment) every PR is re-read and re-run through the
|
||||
// eligibility rules: block labels, approval status on the head, protected approval and an open
|
||||
// release-freeze (which holds the green batch in flight instead of landing it). Then the candidate
|
||||
@@ -291,6 +294,14 @@ const AUTH_FAILURE_RE =
|
||||
*/
|
||||
const RATE_LIMIT_RE = /rate limit/i;
|
||||
|
||||
/**
|
||||
* A GitHub error the client classifies as a rate limit (github.isRateLimited: a 429, or a
|
||||
* primary or secondary rate limit, whose message says "rate limit"); retried no further this tick.
|
||||
*/
|
||||
function isRateLimitError(error) {
|
||||
return error?.status === 429 || RATE_LIMIT_RE.test(String(error?.message ?? ""));
|
||||
}
|
||||
|
||||
function isAuthFailure(error) {
|
||||
if (error?.status === 401) return true;
|
||||
const message = String(error?.message ?? "");
|
||||
@@ -904,9 +915,9 @@ class Tick {
|
||||
};
|
||||
this.event("paused", { detail });
|
||||
this.note(`paused manually: ${detail}`);
|
||||
// Withdrawn by withdrawAbandoned; blessed again only when it lands after the resume. The
|
||||
// landing count goes with the stage: the resume is the operator's fresh start.
|
||||
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight);
|
||||
// Withdrawn by withdrawAbandoned; blessed again only when it lands after the resume. An
|
||||
// operator's pause starts the landing count again (as the resume does).
|
||||
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight, { resetCount: true });
|
||||
} else if (this.action === "resume") {
|
||||
if (!this.s.paused) return this.note("resume: the queue was not paused");
|
||||
const reason = String(this.reason ?? "");
|
||||
@@ -922,6 +933,8 @@ class Tick {
|
||||
}
|
||||
const previous = this.s.paused.reason;
|
||||
this.s.paused = null;
|
||||
// The operator's fresh start: the landing count of the batch in flight starts again.
|
||||
if (this.s.inFlight) this.leaveLandingStage(this.s.inFlight, { resetCount: true });
|
||||
this.event("resumed", { detail: `manual resume (was ${previous})` });
|
||||
this.note(`resumed (was ${previous})`);
|
||||
}
|
||||
@@ -1210,21 +1223,21 @@ class Tick {
|
||||
* landing stage. Nothing is written here: this tick's withdrawAbandoned sees the batch is no
|
||||
* longer kept at the landing stage and turns a `success` land status on its commits into
|
||||
* `failure` (one read per commit, once: later ticks find it stored at "testing"). The landing
|
||||
* count goes too (inFlight.landAttempts): a hold, checks back to pending, a re-run or a pause is
|
||||
* a fresh start, also for a batch the failure path sent back to "testing" with its count kept
|
||||
* (amendFailedTick). Otherwise a no-op at any other stage.
|
||||
* count (inFlight.landAttempts) belongs to the candidate and stays (a hold, checks back to
|
||||
* pending, a re-run, a paused base leaving it as is); only a new candidate and an operator's
|
||||
* pause (`resetCount`) or resume start it again, the latter at any stage.
|
||||
*/
|
||||
leaveLandingStage(f) {
|
||||
leaveLandingStage(f, { resetCount = false } = {}) {
|
||||
if (f.stage !== "landing") {
|
||||
if (f.landAttempts === undefined || this.s.inFlight !== f) return;
|
||||
if (!resetCount || f.landAttempts === undefined || this.s.inFlight !== f) return;
|
||||
const rest = { ...f };
|
||||
delete rest.landAttempts;
|
||||
this.s.inFlight = rest;
|
||||
return this.note(`${f.batchId}: its landing attempts count starts again`);
|
||||
}
|
||||
const rest = { ...f };
|
||||
delete rest.landAttempts;
|
||||
this.s.inFlight = { ...rest, stage: "testing" };
|
||||
const rest = { ...f, stage: "testing", landAttempts: storedLandAttempts(f) };
|
||||
if (resetCount) delete rest.landAttempts;
|
||||
this.s.inFlight = rest;
|
||||
// A dry-run withdraws nothing (withdrawAbandoned returns early): say what would happen.
|
||||
const goes = this.dryRun ? "would go" : "goes";
|
||||
this.note(`${f.batchId}: held at the landing stage, back to testing (its land status ${goes})`);
|
||||
@@ -1258,13 +1271,18 @@ class Tick {
|
||||
const f = this.s.inFlight;
|
||||
if (f?.stage !== "landing" || this.config.landing.mode !== "merge-ff") return;
|
||||
let tip;
|
||||
try {
|
||||
tip = await this.gh.getBranchSha(this.base);
|
||||
} catch (error) {
|
||||
// Unknown: the batch is treated as not landed (withdrawn if this tick leaves it), the
|
||||
// security-first choice of settleFailure too; the tick goes on (a pause is still applied).
|
||||
this.note(`base tip unreadable, ${f.batchId} taken as not landed: ${error.message}`);
|
||||
return;
|
||||
for (let read = 1; ; read++) {
|
||||
try {
|
||||
tip = await this.gh.getBranchSha(this.base);
|
||||
break;
|
||||
} catch (error) {
|
||||
// Two reads: the fall-back below can write `failure` on a tip that did land.
|
||||
if (read < 2) continue;
|
||||
// Unknown: the batch is treated as not landed (withdrawn if this tick leaves it), the
|
||||
// security-first choice of settleFailure too; the tick goes on (a pause is still applied).
|
||||
this.note(`base tip unreadable, ${f.batchId} taken as not landed: ${error.message}`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
this.tipChecked = true;
|
||||
if (tip !== f.candidateSha) return;
|
||||
@@ -1377,8 +1395,9 @@ class Tick {
|
||||
// Earlier ticks blessed this batch and none of them reached a verdict (stored at the landing
|
||||
// stage, or sent back to testing by main()'s failure path with its count kept). After
|
||||
// LANDING_ATTEMPTS_LIMIT of them it is not blessed again: pause and say so.
|
||||
// Live only: shadow blesses nothing, so a count carried from live mode never pauses it.
|
||||
const landAttempts = storedLandAttempts(f);
|
||||
if (landAttempts >= LANDING_ATTEMPTS_LIMIT) {
|
||||
if (live && landAttempts >= LANDING_ATTEMPTS_LIMIT) {
|
||||
const lastError = this.s.history.findLast((e) => e.type === "error")?.detail || "unknown";
|
||||
return this.pauseLanding(
|
||||
f,
|
||||
@@ -1764,8 +1783,9 @@ class Tick {
|
||||
/**
|
||||
* End of a tick: the withdrawals GitHub refused this tick while landWithdrawals was full
|
||||
* (storeWithdrawal: kept in the overflow, or nowhere) are recorded by the pure
|
||||
* recordFullWithdrawals: a base that is not paused pauses with reason `error`, a paused one keeps
|
||||
* its pause, and an `error` event says which commits. A new pause leaves a landing stage (a
|
||||
* recordFullWithdrawals: a base that is not paused (or paused as base-red) pauses with reason
|
||||
* `error`, one paused for another reason keeps its reason (a commit kept nowhere is appended to
|
||||
* its detail), and an `error` event says which commits. A new pause leaves a landing stage (a
|
||||
* paused base holds no blessed batch), so withdrawAbandoned runs once more; a refusal there is
|
||||
* recorded the same way.
|
||||
*/
|
||||
@@ -1815,16 +1835,18 @@ class Tick {
|
||||
* Keep a `failure` GitHub refused to write on `sha` for the next ticks (retryWithdrawals), by
|
||||
* the pure storeWithdrawalIn: one entry per SHA, in state.landWithdrawals while it has room,
|
||||
* else in state.landWithdrawalsOverflow. Nothing is ever dropped. A refusal that did not go to
|
||||
* the main list (overflow, or nowhere when both are full) is held in this.fullRefusals for the
|
||||
* end of the tick (recordIfFull). Every refusal is also kept in this.refused for main()'s
|
||||
* failure path.
|
||||
* the main list as a new entry (overflow, or nowhere when both are full) is held in
|
||||
* this.fullRefusals for the end of the tick (recordIfFull). Every refusal is also kept in
|
||||
* this.refused for main()'s failure path.
|
||||
*/
|
||||
storeWithdrawal(f, sha, why) {
|
||||
const refusal = { sha, batchId: f.batchId, candidateSha: f.candidateSha, why, at: this.now() };
|
||||
this.refused.push(refusal);
|
||||
const { state, list } = storeWithdrawalIn(this.s, refusal);
|
||||
const result = storeWithdrawalIn(this.s, refusal);
|
||||
const { state, list } = result;
|
||||
for (const field of WITHDRAWAL_LISTS) this.setWithdrawalList(field, state[field] ?? []);
|
||||
if (list === "landWithdrawals") return;
|
||||
this.alertAttempts(result.entry, list);
|
||||
if (!isFullListRefusal(result)) return;
|
||||
this.fullRefusals.push({ ...refusal, list });
|
||||
this.note(
|
||||
`${f.batchId}: withdrawal of ${head12(sha)} ` +
|
||||
@@ -1871,24 +1893,39 @@ class Tick {
|
||||
if (this.dryRun) return;
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
const stored = this.s[field] ?? [];
|
||||
if (stored.length) this.setWithdrawalList(field, await this.retryList(stored));
|
||||
if (!stored.length) continue;
|
||||
const { kept, stopped } = await this.retryList(stored, field);
|
||||
this.setWithdrawalList(field, kept);
|
||||
if (stopped) return; // a rate limit: the rest of both lists waits for the next tick
|
||||
}
|
||||
}
|
||||
|
||||
/** retryWithdrawals for one list: returns the entries kept (each with its attempts counted). */
|
||||
async retryList(stored) {
|
||||
/**
|
||||
* retryWithdrawals for one list: returns the entries kept (each failed one with its attempts
|
||||
* counted; an entry that reaches WITHDRAWAL_ALERT_ATTEMPTS records one `error` event). A read or
|
||||
* a write that fails with a rate limit (isRateLimitError) stops the list there: the entries
|
||||
* after it are kept as they are, and `stopped` tells the caller to leave the next list too.
|
||||
*/
|
||||
async retryList(stored, field) {
|
||||
const { statusContext } = this.config.landing;
|
||||
const kept = [];
|
||||
for (const w of stored) {
|
||||
const failed = (w, error) => {
|
||||
const next = { ...w, attempts: w.attempts + 1 };
|
||||
kept.push(next);
|
||||
this.alertAttempts(next, field);
|
||||
return isRateLimitError(error);
|
||||
};
|
||||
for (const [i, w] of stored.entries()) {
|
||||
let newest;
|
||||
try {
|
||||
newest = await this.newestAppLandStatus(w.sha);
|
||||
} catch (error) {
|
||||
kept.push({ ...w, attempts: w.attempts + 1 });
|
||||
const limited = failed(w, error);
|
||||
this.note(
|
||||
`could not read the land status of ${head12(w.sha)} (stored withdrawal of ` +
|
||||
`${w.batchId}, ${w.attempts + 1} failed attempts): ${error.message}`
|
||||
);
|
||||
if (limited) return this.rateLimited(kept, stored.slice(i + 1));
|
||||
continue;
|
||||
}
|
||||
if (newest?.state !== "success") {
|
||||
@@ -1912,14 +1949,32 @@ class Tick {
|
||||
);
|
||||
this.retired.push(w.sha);
|
||||
} catch (error) {
|
||||
kept.push({ ...w, attempts: w.attempts + 1 });
|
||||
const limited = failed(w, error);
|
||||
this.note(
|
||||
`${w.batchId}: withdrawal of ${statusContext} on ${head12(w.sha)} refused again ` +
|
||||
`(${w.attempts + 1} failed attempts): ${String(error?.message ?? error).slice(0, 200)}`
|
||||
);
|
||||
if (limited) return this.rateLimited(kept, stored.slice(i + 1));
|
||||
}
|
||||
}
|
||||
return kept;
|
||||
return { kept, stopped: false };
|
||||
}
|
||||
|
||||
/** retryList stopped by a rate limit: what it kept, then the untouched rest of the list. */
|
||||
rateLimited(kept, rest) {
|
||||
this.note("stored withdrawals: rate limited, the rest waits for the next tick");
|
||||
return { kept: [...kept, ...rest], stopped: true };
|
||||
}
|
||||
|
||||
/** One `error` event when a stored withdrawal's attempts reach WITHDRAWAL_ALERT_ATTEMPTS. */
|
||||
alertAttempts(entry, field) {
|
||||
const before = this.s.history.length;
|
||||
this.s = withAttemptsAlert(this.s, entry, field, {
|
||||
at: this.now(),
|
||||
url: this.deps.runUrl ?? undefined,
|
||||
statusContext: this.config.landing.statusContext,
|
||||
});
|
||||
if (this.s.history.length !== before) this.note(this.s.history.at(-1).detail);
|
||||
}
|
||||
|
||||
async afterLanded(f, sha) {
|
||||
@@ -2542,7 +2597,8 @@ class Tick {
|
||||
const startedAt = pushed === "existed" ? this.retryCutoff() : pushedAt;
|
||||
const rest = { ...f };
|
||||
delete rest.timedOutRuns; // carried only while every run so far timed out
|
||||
delete rest.landAttempts; // counted only at the landing stage, which a retry leaves
|
||||
// landAttempts stays: the same candidate SHA is tested again (only a new candidate or an
|
||||
// operator's pause or resume starts the count again).
|
||||
delete rest.rerun; // a new ref: its runs are not the re-run's (the retry is never re-run)
|
||||
delete rest.freezeHold; // a new run, a new startedAt: the in-flight clock starts again
|
||||
this.s.inFlight = {
|
||||
@@ -2552,6 +2608,7 @@ class Tick {
|
||||
attempts,
|
||||
startedAt,
|
||||
stage: "testing",
|
||||
...(storedLandAttempts(f) > 0 ? { landAttempts: storedLandAttempts(f) } : {}),
|
||||
...(everyRunTimedOut ? { timedOutRuns: attempts } : {}),
|
||||
};
|
||||
this.s.counters.ciRuns += 1;
|
||||
@@ -3572,9 +3629,9 @@ function withWithdrawalList(state, field, list) {
|
||||
* state.landWithdrawalsOverflow (same shape, same limit). Nothing is ever dropped to make room.
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string, candidateSha: string, why: string, at: string}} refusal
|
||||
* @returns {{state: QueueState, list: string|null}} the list that holds it now, or null when both
|
||||
* are full (the state is then unchanged and the caller names the commit,
|
||||
* recordFullWithdrawals)
|
||||
* @returns {{state: QueueState, list: string|null, added: boolean, entry: object|null}} the list
|
||||
* that holds it now (null when both are full: the state is then unchanged and the caller names
|
||||
* the commit, recordFullWithdrawals), whether that is a new entry, and the entry
|
||||
*/
|
||||
export function storeWithdrawalIn(state, refusal) {
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
@@ -3582,25 +3639,75 @@ export function storeWithdrawalIn(state, refusal) {
|
||||
const i = list.findIndex((w) => w.sha === refusal.sha);
|
||||
if (i < 0) continue;
|
||||
list[i] = { ...list[i], why: refusal.why, attempts: list[i].attempts + 1 };
|
||||
return { state: withWithdrawalList(state, field, list), list: field };
|
||||
return {
|
||||
state: withWithdrawalList(state, field, list),
|
||||
list: field,
|
||||
added: false,
|
||||
entry: list[i],
|
||||
};
|
||||
}
|
||||
const { sha, batchId, candidateSha, why, at } = refusal;
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
const list = state[field] ?? [];
|
||||
if (list.length >= LAND_WITHDRAWALS_LIMIT) continue;
|
||||
const entry = { sha, batchId, candidateSha, why, since: at, attempts: 1 };
|
||||
return { state: withWithdrawalList(state, field, [...list, entry]), list: field };
|
||||
return {
|
||||
state: withWithdrawalList(state, field, [...list, entry]),
|
||||
list: field,
|
||||
added: true,
|
||||
entry,
|
||||
};
|
||||
}
|
||||
return { state, list: null };
|
||||
return { state, list: null, added: false, entry: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether storing a refusal (storeWithdrawalIn's result) is the full-list condition: a NEW entry
|
||||
* in the overflow, or none at all (both lists full). A refusal of a SHA the overflow already holds
|
||||
* is not: the main list may have room again by then.
|
||||
*/
|
||||
function isFullListRefusal({ list, added }) {
|
||||
return list === null || (list === "landWithdrawalsOverflow" && added);
|
||||
}
|
||||
|
||||
/**
|
||||
* Failed attempts of one stored withdrawal after which one `error` event says it is still not
|
||||
* withdrawn (about a day of five-minute ticks). The queue keeps retrying: an entry never expires.
|
||||
*/
|
||||
export const WITHDRAWAL_ALERT_ATTEMPTS = 288;
|
||||
|
||||
/**
|
||||
* `state` plus the `error` event of a stored withdrawal whose attempts just reached
|
||||
* WITHDRAWAL_ALERT_ATTEMPTS (pure); `state` unchanged at any other count.
|
||||
*/
|
||||
function withAttemptsAlert(state, entry, field, { at, url, statusContext }) {
|
||||
if (entry?.attempts !== WITHDRAWAL_ALERT_ATTEMPTS) return state;
|
||||
const detail =
|
||||
`${entry.batchId}: the withdrawal of ${statusContext} on ${entry.sha} (${field}) has ` +
|
||||
`failed ${WITHDRAWAL_ALERT_ATTEMPTS} times since ${entry.since}; the queue keeps retrying it`;
|
||||
return appendEvent(
|
||||
state,
|
||||
stripUndefined({ at, type: "error", batchId: entry.batchId, sha: entry.sha, detail, url })
|
||||
);
|
||||
}
|
||||
|
||||
/** Start of the pause detail and of the `error` event of a full landWithdrawals list. */
|
||||
export const WITHDRAWALS_FULL_DETAIL = "landWithdrawals is full";
|
||||
|
||||
/** Words that introduce, in paused.detail, the commits a full pair of lists could not keep. */
|
||||
export const KEPT_NOWHERE = "Kept nowhere (both lists full)";
|
||||
|
||||
/**
|
||||
* Landings of `f` that ended without a verdict, as stored: `landAttempts` at any stage (the
|
||||
* failure path keeps it when it sends a batch back to "testing"); a batch at the landing stage of
|
||||
* a document written before the count counts as 1.
|
||||
* Pause reasons a full withdrawal list leaves as they are: none of them lifts by itself and
|
||||
* nothing is blessed under one. A `base-red` pause does both (the base-repair lane lands under it,
|
||||
* maybeAutoResume lifts it when the tip moves), so the full-list pause replaces it.
|
||||
*/
|
||||
const KEPT_PAUSE_REASONS = Object.freeze(["manual", "error", "infra", "tree-divergence"]);
|
||||
|
||||
/**
|
||||
* Landings of `f` that ended without a verdict, as stored: `landAttempts` at any stage (the count
|
||||
* belongs to the candidate, not to the stage); a batch at the landing stage of a document written
|
||||
* before the count counts as 1.
|
||||
* @param {InFlight} f
|
||||
*/
|
||||
function storedLandAttempts(f) {
|
||||
@@ -3610,31 +3717,49 @@ function storedLandAttempts(f) {
|
||||
|
||||
/**
|
||||
* `state` with its in-flight batch at stage "testing" when it was at "landing" (pure); every
|
||||
* other field stays. `keepCount` (main()'s failure path) keeps `landAttempts`, so the attempts
|
||||
* limit still counts the landings that ended without a verdict; a pause resets it, as
|
||||
* Tick.leaveLandingStage does.
|
||||
* other field stays, its landing count included (written out: a document of the code before the
|
||||
* count has none at the landing stage, where it counts as 1). Only a new candidate and an
|
||||
* operator's pause or resume start the count again.
|
||||
*/
|
||||
function leaveLandingIn(state, { keepCount = false } = {}) {
|
||||
function leaveLandingIn(state) {
|
||||
const f = state.inFlight;
|
||||
if (f?.stage !== "landing") return state;
|
||||
const rest = { ...f, stage: "testing" };
|
||||
if (!keepCount) delete rest.landAttempts;
|
||||
return { ...state, inFlight: rest };
|
||||
return { ...state, inFlight: { ...f, stage: "testing", landAttempts: storedLandAttempts(f) } };
|
||||
}
|
||||
|
||||
/** "<sha> (<batchId>)", comma-separated. */
|
||||
const namedCommits = (list) => list.map((r) => `${r.sha} (${r.batchId})`).join(", ");
|
||||
|
||||
/**
|
||||
* The commits of `list`, named in full, as many as fit in `room` characters, then
|
||||
* "and N more, named in paused.detail".
|
||||
*/
|
||||
function namedWithin(list, room) {
|
||||
for (let n = list.length; n > 0; n--) {
|
||||
const more = n < list.length ? `, and ${list.length - n} more, named in paused.detail` : "";
|
||||
const text = namedCommits(list.slice(0, n)) + more;
|
||||
if (text.length <= room) return text;
|
||||
}
|
||||
return `${list.length} commit(s), named in paused.detail`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record (pure) the refusals of a tick that found state.landWithdrawals full: those it kept in
|
||||
* state.landWithdrawalsOverflow (`list` "landWithdrawalsOverflow") and those it could keep nowhere
|
||||
* (`list` null, both lists full). A full list means GitHub refuses the App's status writes, so the
|
||||
* queue must not go on blessing:
|
||||
* - a base that is not paused is paused with reason `error`; the detail says how many entries
|
||||
* each list holds and names in full each commit kept nowhere (the only record of it, with the
|
||||
* event below); the batch in flight leaves the landing stage (its count reset, as for any
|
||||
* pause);
|
||||
* - a base already paused keeps its pause untouched (reason, detail, since), except a pause of
|
||||
* this kind, which gets the commits kept nowhere appended;
|
||||
* Record (pure) the refusals of a tick that found state.landWithdrawals full: those it kept as new
|
||||
* entries of state.landWithdrawalsOverflow (`list` "landWithdrawalsOverflow") and those it could
|
||||
* keep nowhere (`list` null, both lists full). A full list means GitHub refuses the App's status
|
||||
* writes, so the queue must not go on blessing:
|
||||
* - a base that is not paused, or paused as `base-red` (which lifts by itself and blesses its
|
||||
* repair lane), is paused with reason `error`: the detail says how many entries each list
|
||||
* holds, names in full each commit kept nowhere and, for a replaced base-red pause, says so in
|
||||
* one clause; the batch in flight leaves the landing stage (its count kept);
|
||||
* - a base paused for a reason of KEPT_PAUSE_REASONS keeps its reason and `since`; a commit kept
|
||||
* nowhere is appended to its detail (`Kept nowhere (both lists full): <sha> (<batchId>)`), so
|
||||
* paused.detail names every such commit, whatever the pause;
|
||||
* - in every case one `error` event says the same, its `sha` the first commit kept nowhere, else
|
||||
* the first kept in the overflow.
|
||||
* the first kept in the overflow; its detail is cut at 500 characters, so it names the commits
|
||||
* kept nowhere that fit and how many more paused.detail holds.
|
||||
* paused.detail has no length bound in validateState: every commit fits there (the dashboard and
|
||||
* the CLI show a cut copy).
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string, list: string|null}[]} refusals
|
||||
* @param {{at: string, url?: string, statusContext: string}} options
|
||||
@@ -3645,44 +3770,49 @@ export function recordFullWithdrawals(state, refusals, { at, url, statusContext
|
||||
const all = [...bySha.values()];
|
||||
if (!all.length) return state;
|
||||
const nowhere = all.filter((r) => r.list === null);
|
||||
const named = (list) => list.map((r) => `${r.sha} (${r.batchId})`).join(", ");
|
||||
const counts =
|
||||
`${(state.landWithdrawals ?? []).length} of ${LAND_WITHDRAWALS_LIMIT}; ` +
|
||||
`landWithdrawalsOverflow holds ${(state.landWithdrawalsOverflow ?? []).length} of ` +
|
||||
`${LAND_WITHDRAWALS_LIMIT}`;
|
||||
const lost = nowhere.length
|
||||
? `. Kept nowhere (both lists full), the only record of them: ${named(nowhere)}; check ` +
|
||||
`their ${statusContext} by hand (docs/ops/OMNI_QUEUE.md, Incidents)`
|
||||
: "";
|
||||
const paused = state.paused;
|
||||
const ours =
|
||||
paused?.reason === "error" && String(paused.detail ?? "").startsWith(WITHDRAWALS_FULL_DETAIL);
|
||||
const kept = paused && KEPT_PAUSE_REASONS.includes(paused.reason);
|
||||
let next = state;
|
||||
if (!paused) {
|
||||
if (!kept) {
|
||||
const lost = nowhere.length
|
||||
? `. ${KEPT_NOWHERE}: ${namedCommits(nowhere)}; check their ${statusContext} by hand ` +
|
||||
"(docs/ops/OMNI_QUEUE.md, Incidents)"
|
||||
: "";
|
||||
const replaced = paused
|
||||
? `; the base was also paused as ${paused.reason}: ${String(paused.detail).slice(0, 200)}`
|
||||
: "";
|
||||
const detail =
|
||||
`${WITHDRAWALS_FULL_DETAIL} (${counts}): GitHub refused the App's ${statusContext} ` +
|
||||
"failure again; every refused withdrawal is kept and retried on every tick. Find out why " +
|
||||
"GitHub refuses the App's status writes, wait until both lists are empty, then resume" +
|
||||
lost;
|
||||
lost +
|
||||
replaced;
|
||||
next = leaveLandingIn({
|
||||
...state,
|
||||
paused: { reason: "error", since: at, baseSha: null, detail, issue: null },
|
||||
});
|
||||
} else if (ours) {
|
||||
const fresh = nowhere.filter((r) => !paused.detail.includes(r.sha));
|
||||
} else {
|
||||
const fresh = nowhere.filter((r) => !String(paused.detail).includes(r.sha));
|
||||
if (fresh.length) {
|
||||
const detail = `${paused.detail}; also kept nowhere: ${named(fresh)}`;
|
||||
const detail = `${paused.detail}; ${KEPT_NOWHERE}: ${namedCommits(fresh)}`;
|
||||
next = { ...state, paused: { ...paused, detail } };
|
||||
}
|
||||
}
|
||||
const overflowed = all.filter((r) => r.list !== null).map((r) => head12(r.sha));
|
||||
const detail =
|
||||
const head =
|
||||
`${WITHDRAWALS_FULL_DETAIL} (${counts})` +
|
||||
(overflowed.length
|
||||
? `: refused withdrawals kept in the overflow: ${overflowed.join(", ")}`
|
||||
: "") +
|
||||
(nowhere.length ? `; kept nowhere: ${named(nowhere)}` : "") +
|
||||
(paused && !ours ? `; the base stays paused (${paused.reason})` : "");
|
||||
(kept ? `; the base stays paused (${paused.reason})` : "") +
|
||||
(paused && !kept ? `; the ${paused.reason} pause was replaced` : "");
|
||||
const detail = nowhere.length
|
||||
? `${head}; kept nowhere: ${namedWithin(nowhere, 500 - head.length - 16)}`
|
||||
: head;
|
||||
const first = nowhere[0] ?? all[0];
|
||||
return appendEvent(
|
||||
next,
|
||||
@@ -3701,8 +3831,8 @@ export function recordFullWithdrawals(state, refusals, { at, url, statusContext
|
||||
* Apply to the stored state what a tick that threw left for it (failedTickRecord), pure: drop
|
||||
* the stored withdrawals it settled, take the batch it withdrew off the landing stage when the
|
||||
* stored batch is still that one at that stage (its landing count kept), store the withdrawals
|
||||
* GitHub refused (and record a full list, recordFullWithdrawals). A null record
|
||||
* returns `state` unchanged.
|
||||
* GitHub refused (with the alert of an entry that reaches WITHDRAWAL_ALERT_ATTEMPTS, and record a
|
||||
* full list, recordFullWithdrawals). A null record returns `state` unchanged.
|
||||
* @param {QueueState} state
|
||||
* @param {ReturnType<typeof failedTickRecord>} record
|
||||
* @param {{at: string, url?: string, statusContext: string}} options
|
||||
@@ -3728,13 +3858,13 @@ export function amendFailedTick(state, record, options) {
|
||||
f.batchId === leave.batchId &&
|
||||
f.candidateSha === leave.candidateSha
|
||||
) {
|
||||
next = leaveLandingIn(next, { keepCount: true });
|
||||
next = leaveLandingIn(next);
|
||||
}
|
||||
const full = [];
|
||||
for (const refusal of record.withdrawals ?? []) {
|
||||
const result = storeWithdrawalIn(next, refusal);
|
||||
next = result.state;
|
||||
if (result.list !== "landWithdrawals") full.push({ ...refusal, list: result.list });
|
||||
next = withAttemptsAlert(result.state, result.entry, result.list, options);
|
||||
if (isFullListRefusal(result)) full.push({ ...refusal, list: result.list });
|
||||
}
|
||||
return full.length ? recordFullWithdrawals(next, full, options) : next;
|
||||
}
|
||||
|
||||
@@ -29,11 +29,13 @@ import { NO_HEARTBEAT_LINE, assessHealth } from "../../../scripts/merge-queue/da
|
||||
import {
|
||||
GH_WRITE_METHODS,
|
||||
LANDING_ATTEMPTS_LIMIT,
|
||||
WITHDRAWAL_ALERT_ATTEMPTS,
|
||||
RERUN_START_MINUTES,
|
||||
STATE_HEARTBEAT_MINUTES,
|
||||
BlessRefusedError,
|
||||
StateConflictError,
|
||||
TICK_ACTIONS,
|
||||
amendFailedTick,
|
||||
createTickDeps,
|
||||
discoverBases,
|
||||
failedTickRecord,
|
||||
@@ -41,6 +43,7 @@ import {
|
||||
main,
|
||||
parseArgs,
|
||||
readOnlyGitHub,
|
||||
recordFullWithdrawals,
|
||||
recordTickFailure,
|
||||
runTick,
|
||||
WITHDRAWALS_FULL_DETAIL,
|
||||
@@ -1573,7 +1576,7 @@ test("a red batch at the landing stage leaves it before its failed jobs are re-r
|
||||
const state = await h.state();
|
||||
assert.equal(state.history.at(-1).type, "flaky-rerun");
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
|
||||
assert.equal(state.inFlight.rerun.ref, f.candidateRef);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure", "its land status is withdrawn");
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
@@ -2015,7 +2018,7 @@ test("a red batch at the landing stage loses its land status on the first red ti
|
||||
assert.deepEqual(types(state), ["batch-started"], "it only waits");
|
||||
assert.deepEqual(rerunCalls(h), []);
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
|
||||
assert.equal("rerun" in state.inFlight, false);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure", "its land status is withdrawn");
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
@@ -3142,7 +3145,7 @@ test("a landing-stage batch whose checks went back to pending is withdrawn and r
|
||||
let state = await h.state();
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure", "not blessed while its checks run");
|
||||
assert.deepEqual(statusStates(h), ["failure"]);
|
||||
@@ -3160,8 +3163,8 @@ test("a landing-stage batch whose checks went back to pending is withdrawn and r
|
||||
.filter((message) => message.startsWith(`omni-queue: landing ${f.batchId} `));
|
||||
assert.deepEqual(
|
||||
marker,
|
||||
[`omni-queue: landing ${f.batchId} on ${BASE}`],
|
||||
"the landing stage is stored again before the blessing, as attempt 1: a hold resets the count"
|
||||
[`omni-queue: landing ${f.batchId} on ${BASE} (attempt 2)`],
|
||||
"the landing stage is stored again before the blessing, as attempt 2: the count is the candidate's (W3)"
|
||||
);
|
||||
});
|
||||
|
||||
@@ -3607,7 +3610,7 @@ test("live → shadow on a paused base: a batch left at the landing stage loses
|
||||
assert.equal(state.paused?.reason, "infra", "still paused");
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
|
||||
assert.equal(state.inFlight.stage, "testing", "shadow keeps no batch at the landing stage");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
});
|
||||
|
||||
@@ -4001,9 +4004,9 @@ for (const blessed of [false, true]) {
|
||||
assert.match(held[0].detail, /^base-frozen: freeze on release\/v9\.9\.9/);
|
||||
assert.equal(state.history.filter((e) => e.type === "checks-green").length, 1);
|
||||
assert.deepEqual(h.writesOf("removeLabel"), []);
|
||||
// Held = not at the landing stage: a blessing is withdrawn, and the landing count goes too.
|
||||
// Held = not at the landing stage: a blessing is withdrawn; the landing count stays (W3).
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, blessed ? 1 : undefined, "the candidate's count");
|
||||
const withdrawn = blessed ? [landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })] : [];
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
@@ -7223,7 +7226,7 @@ test("M1 probe: a landing-stage batch whose base moved and whose recomposition p
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
|
||||
assert.equal(state.inFlight.stage, "testing", `tick ${tick}: no landing claimed`);
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the count belongs to the candidate: kept (W3)");
|
||||
assert.equal(state.failedTicks, tick);
|
||||
assert.equal(
|
||||
state.history.findLast((e) => e.type === "error").detail.includes("hung up"),
|
||||
@@ -7729,7 +7732,7 @@ test("V1 (review probe 1): landings that throw, alternating with ticks that thro
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
});
|
||||
|
||||
test("V1: a freeze hold between two failing landings still resets the count", async () => {
|
||||
test("W3 (was V1): a freeze hold between two failing landings keeps the count", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await greenBatch(h, [1]);
|
||||
const getPull = h.gh.getPull;
|
||||
@@ -7744,12 +7747,12 @@ test("V1: a freeze hold between two failing landings still resets the count", as
|
||||
await h.tick();
|
||||
let state = await h.state();
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false, "the hold resets it");
|
||||
assert.equal(state.inFlight.landAttempts, 1, "the hold keeps it");
|
||||
freeze.state = "closed";
|
||||
h.clock.advance(10);
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
state = await h.state();
|
||||
assert.equal(state.inFlight.landAttempts, 1, "counted from 1 again");
|
||||
assert.equal(state.inFlight.landAttempts, 2, "counted on");
|
||||
});
|
||||
|
||||
test("V4: a batch loaded at the landing stage and recomposed has its status withdrawn before the queue snapshot", async () => {
|
||||
@@ -7872,7 +7875,8 @@ test("V5: the failure path leaves a batch whose candidate is the base tip at the
|
||||
const getBranchSha = h.gh.getBranchSha;
|
||||
let reads = 0;
|
||||
h.gh.getBranchSha = async (ref) => {
|
||||
if (ref === BASE && reads++ === 0) throw new Error("GET /git/ref → 502: Bad Gateway");
|
||||
// both reads of recordEarlierLanding fail (W4); the failure path's own read answers
|
||||
if (ref === BASE && reads++ < 2) throw new Error("GET /git/ref → 502: Bad Gateway");
|
||||
return getBranchSha(ref);
|
||||
};
|
||||
const listCheckRuns = h.gh.listCheckRuns;
|
||||
@@ -7889,3 +7893,288 @@ test("V5: the failure path leaves a batch whose candidate is the base tip at the
|
||||
assert.equal(state.history.filter((e) => e.type === "landed").length, 1);
|
||||
assert.deepEqual(statusCalls(h), []);
|
||||
});
|
||||
|
||||
// ─────────── Task 13b fix round 2 (W1 to W4) ───────────
|
||||
|
||||
/** 20 + 20 stored withdrawals of App-blessed commits: both lists full. */
|
||||
async function bothListsFull(h) {
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
const spill = Array.from({ length: 20 }, (_, i) => fakeSha(`spill-${i}`));
|
||||
for (const sha of [...old, ...spill]) setLand(h, sha, "success");
|
||||
const { state, version } = await h.store.load(BASE);
|
||||
state.landWithdrawals = old.map((sha) => storedWithdrawal(sha));
|
||||
state.landWithdrawalsOverflow = spill.map((sha) => storedWithdrawal(sha));
|
||||
assert.equal((await h.store.save(BASE, state, version, "seed: both lists full")).ok, true);
|
||||
return { old, spill };
|
||||
}
|
||||
|
||||
test("W1 (re-review RR-8): a base paused by hand, both lists full: the commit kept nowhere is named in that pause's detail, its reason and since untouched", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await bothListsFull(h);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]);
|
||||
refuseWithdrawals(h, () => true);
|
||||
const at = h.clock.now();
|
||||
await h.tick({ action: "pause", reason: "before the mode change" });
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused.reason, "manual");
|
||||
assert.equal(state.paused.since, at, "the operator's pause time");
|
||||
assert.ok(state.paused.detail.startsWith("before the mode change"), state.paused.detail);
|
||||
assert.ok(
|
||||
state.paused.detail.includes(
|
||||
`Kept nowhere (both lists full): ${f.candidateSha} (${f.batchId})`
|
||||
),
|
||||
state.paused.detail
|
||||
);
|
||||
const event = state.history.findLast((e) => e.type === "error");
|
||||
assert.equal(event.sha, f.candidateSha);
|
||||
});
|
||||
|
||||
test("W1: more commits kept nowhere than an event detail holds: all in paused.detail, the event names what fits and how many more", () => {
|
||||
const heads = Array.from({ length: 12 }, (_, i) => ({
|
||||
sha: fakeSha(`head-nowhere-${i}`),
|
||||
batchId: "b12",
|
||||
list: null,
|
||||
}));
|
||||
const paused = {
|
||||
reason: "manual",
|
||||
since: "2026-11-13T11:00:00.000Z",
|
||||
baseSha: null,
|
||||
detail: "op",
|
||||
issue: null,
|
||||
};
|
||||
const next = recordFullWithdrawals({ ...initialState(BASE), paused }, heads, {
|
||||
at: "2026-11-13T12:00:00.000Z",
|
||||
statusContext: "omni-queue/land",
|
||||
});
|
||||
assert.equal(next.paused.reason, "manual");
|
||||
assert.equal(next.paused.since, paused.since);
|
||||
for (const head of heads) assert.ok(next.paused.detail.includes(head.sha), head.sha);
|
||||
const event = next.history.at(-1);
|
||||
assert.ok(event.detail.length <= 500);
|
||||
const named = heads.filter((head) => event.detail.includes(head.sha)).length;
|
||||
assert.ok(named >= 1 && named < 12, `named ${named}`);
|
||||
assert.ok(event.detail.includes(`and ${12 - named} more, named in paused.detail`), event.detail);
|
||||
});
|
||||
|
||||
test("W2: a base-red pause with a repair in flight is replaced by the full-list pause: nothing is blessed under it and it does not lift when the tip moves", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { baseRepairLabel } = h.config.lanes;
|
||||
await baseRedPause(h, [1, 2]);
|
||||
h.addPr(3);
|
||||
h.addLane(3, baseRepairLabel);
|
||||
h.world.repairs.add(3);
|
||||
await h.tick();
|
||||
const repair = (await h.state()).inFlight;
|
||||
assert.equal(repair.lane, "base-repair");
|
||||
h.runCi(repair.candidateSha);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
for (const sha of old) setLand(h, sha, "success");
|
||||
await storeWithdrawals(
|
||||
h,
|
||||
old.map((sha) => storedWithdrawal(sha))
|
||||
);
|
||||
h.world.baseMovesOnLand = fakeSha("manual-merge"); // the repair's landing is refused
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
let state = await h.state();
|
||||
assert.equal(state.paused.reason, "error", "the base-red pause is replaced");
|
||||
assert.ok(state.paused.detail.startsWith(WITHDRAWALS_FULL_DETAIL), state.paused.detail);
|
||||
assert.match(state.paused.detail, /also paused as base-red/);
|
||||
const blessings = statusStates(h).filter((s) => s === "success").length;
|
||||
|
||||
const next = state.inFlight;
|
||||
if (next) h.runCi(next.candidateSha);
|
||||
h.gh.branches.set(BASE, fakeSha("tip-moved-again"));
|
||||
for (let i = 0; i < 2; i++) {
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
}
|
||||
state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error", "the tip moved: still paused");
|
||||
assert.equal(statusStates(h).filter((s) => s === "success").length, blessings, "nothing blessed");
|
||||
});
|
||||
|
||||
test("W3 (re-review RR-2): landings that throw, alternating with checks back to pending, pause at the third landing without a verdict", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
h.world.landError = notAvailable(1);
|
||||
for (let attempt = 1; attempt <= 3; attempt++) {
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
assert.equal((await h.state()).inFlight.landAttempts, attempt);
|
||||
rerunGate(h, f.candidateSha, "in_progress");
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(state.inFlight.stage, "testing", "pending: back to testing");
|
||||
assert.equal(state.inFlight.landAttempts, attempt, "the count is the candidate's: kept");
|
||||
rerunGate(h, f.candidateSha, "completed");
|
||||
}
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.match(state.paused.detail, /ended without a verdict 3 times in a row/);
|
||||
assert.equal(statusStates(h).filter((s) => s === "success").length, 3, "three blessings");
|
||||
});
|
||||
|
||||
test("W3 (re-review RR-10): a shadow tick never pauses on a kept count", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await greenBatch(h, [1]);
|
||||
const getPull = h.gh.getPull;
|
||||
h.world.landError = notAvailable(1);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
h.gh.getPull = getPull;
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
}
|
||||
h.gh.getPull = brokenGetPull();
|
||||
await recordedFailingTick(h, /502/);
|
||||
h.gh.getPull = getPull;
|
||||
assert.equal((await h.state()).inFlight.landAttempts, 3);
|
||||
h.config.mode = "shadow";
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused, null, "shadow blesses nothing, so it never pauses for the limit");
|
||||
assert.ok(types(state).includes("would-land"));
|
||||
});
|
||||
|
||||
test("W3: a recomposition starts the count again", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
h.world.landError = notAvailable(1);
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
h.world.landError = null;
|
||||
h.gh.branches.set(BASE, fakeSha("manual-merge"));
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.notEqual(state.inFlight.candidateSha, f.candidateSha, "recomposed");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
});
|
||||
|
||||
test("W3: an operator's pause, and an operator's resume, start the count again", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await greenBatch(h, [1]);
|
||||
h.world.landError = notAvailable(1);
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
assert.equal("landAttempts" in (await h.state()).inFlight, false, "the pause resets it");
|
||||
// A resume of a pause the operator did not make (here: an infra pause) resets it too.
|
||||
const { state: stored, version } = await h.store.load(BASE);
|
||||
stored.paused = {
|
||||
reason: "infra",
|
||||
since: h.clock.now(),
|
||||
baseSha: null,
|
||||
detail: "x",
|
||||
issue: null,
|
||||
};
|
||||
stored.inFlight = { ...stored.inFlight, landAttempts: 2 };
|
||||
assert.equal((await h.store.save(BASE, stored, version, "seed")).ok, true);
|
||||
h.clock.advance(10);
|
||||
h.world.landError = notAvailable(2);
|
||||
await recordedFailingTick(h, /not available from origin/, { action: "resume" });
|
||||
assert.equal((await h.state()).inFlight.landAttempts, 1, "the resume's own landing is attempt 1");
|
||||
});
|
||||
|
||||
test("W4 (re-review RR-9a): a refusal of a SHA already in the overflow, with room in the main list, is not a full list", () => {
|
||||
const sha = (n) => n.toString(16).padStart(40, "0");
|
||||
const entry = (n) => storedWithdrawal(sha(n), { candidateSha: sha(n) });
|
||||
const state = {
|
||||
...initialState(BASE),
|
||||
landWithdrawals: [entry(1), entry(2)],
|
||||
landWithdrawalsOverflow: [entry(3)],
|
||||
};
|
||||
const refusal = {
|
||||
sha: sha(3),
|
||||
batchId: "b9",
|
||||
candidateSha: sha(3),
|
||||
why: "y",
|
||||
at: "2026-11-13T12:00:00.000Z",
|
||||
};
|
||||
const next = amendFailedTick(
|
||||
state,
|
||||
{ leaveLanding: null, withdrawals: [refusal], retired: [] },
|
||||
{ at: "2026-11-13T12:00:00.000Z", statusContext: "omni-queue/land" }
|
||||
);
|
||||
assert.equal(next.paused, null);
|
||||
assert.equal(next.landWithdrawalsOverflow[0].attempts, 2);
|
||||
assert.ok(!next.history.some((e) => e.type === "error"));
|
||||
});
|
||||
|
||||
test("W4: stored retries stop at a rate limit: the rest of both lists waits for the next tick", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
const [ok, limited, later, spilled] = ["ok", "limited", "later", "spilled"].map(fakeSha);
|
||||
for (const sha of [ok, limited, later, spilled]) setLand(h, sha, "success");
|
||||
const { state: stored, version } = await h.store.load(BASE);
|
||||
stored.landWithdrawals = [ok, limited, later].map((sha) => storedWithdrawal(sha));
|
||||
stored.landWithdrawalsOverflow = [storedWithdrawal(spilled)];
|
||||
assert.equal((await h.store.save(BASE, stored, version, "seed")).ok, true);
|
||||
const list = h.gh.listCommitStatuses;
|
||||
h.gh.listCommitStatuses = async (sha) => {
|
||||
if (sha === limited) {
|
||||
throw Object.assign(
|
||||
new Error("GET statuses → 403: API rate limit exceeded for installation"),
|
||||
{
|
||||
name: "GitHubError",
|
||||
status: 403,
|
||||
}
|
||||
);
|
||||
}
|
||||
return list(sha);
|
||||
};
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(landStatusReads(h, later), 0, "not read after the rate limit");
|
||||
assert.equal(landStatusReads(h, spilled), 0, "the overflow neither");
|
||||
assert.equal(newestLand(h, ok), "failure", "the one before it was written");
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((w) => [w.sha, w.attempts]),
|
||||
[
|
||||
[limited, 2],
|
||||
[later, 1],
|
||||
]
|
||||
);
|
||||
assert.deepEqual(
|
||||
state.landWithdrawalsOverflow.map((w) => [w.sha, w.attempts]),
|
||||
[[spilled, 1]]
|
||||
);
|
||||
});
|
||||
|
||||
test("W4: one error event when a stored withdrawal reaches WITHDRAWAL_ALERT_ATTEMPTS, not on the ticks after", async () => {
|
||||
assert.equal(WITHDRAWAL_ALERT_ATTEMPTS, 288);
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
const sha = fakeSha("stubborn");
|
||||
setLand(h, sha, "success");
|
||||
await storeWithdrawals(h, [storedWithdrawal(sha, { attempts: 287 })]);
|
||||
refuseWithdrawals(h, () => true);
|
||||
const alerts = (state) =>
|
||||
state.history.filter((e) => e.type === "error" && e.sha === sha && /288/.test(e.detail ?? ""));
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
let state = await h.state();
|
||||
assert.equal(state.landWithdrawals[0].attempts, 288);
|
||||
assert.equal(alerts(state).length, 1);
|
||||
assert.match(alerts(state)[0].detail, /landWithdrawals/);
|
||||
assert.match(alerts(state)[0].detail, /keeps retrying/);
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
state = await h.state();
|
||||
assert.equal(state.landWithdrawals[0].attempts, 289);
|
||||
assert.equal(alerts(state).length, 1, "once, at the crossing");
|
||||
});
|
||||
|
||||
test("W4: the tip read of recordEarlierLanding is tried twice before falling back", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await landedButNotSaved(h, f);
|
||||
const getBranchSha = h.gh.getBranchSha;
|
||||
let reads = 0;
|
||||
h.gh.getBranchSha = async (ref) => {
|
||||
if (ref === BASE && reads++ === 0) throw new Error("GET /git/ref → 502: Bad Gateway");
|
||||
return getBranchSha(ref);
|
||||
};
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
const state = await h.state();
|
||||
assert.deepEqual(statusCalls(h), [], "no failure on the landed tip");
|
||||
assert.equal(state.history.filter((e) => e.type === "landed").length, 1);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user