mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-11-18 07:58:27 +00:00
fix(ci): close the omni-queue withdrawal gaps left for live mode
Task 13b: the paths where a land status (`omni-queue/land` success) could
be left standing on a batch that did not land, recorded as "before mode
live".
M1, a tick that throws late. run() now catches what runLoaded() throws:
Tick.settleFailure withdraws the land status of the batch the tick found
at the landing stage (forced withdrawAbandoned) unless the tick landed it,
its candidate is the base tip, or the tick made its own landing attempt
(that attempt withdrew its own blessing, and its landing stage with
landAttempts is the attempts bound's record). It never throws and never
replaces the error; it skips a dry-run and a lost compare-and-swap. What
the stored state must learn goes to main() keyed by the error
(failedTickRecord) and recordTickFailure applies it in the same save as
the `error` event (amendFailedTick): the withdrawn batch back to testing,
the refused withdrawals stored, the settled ones dropped. A shadow tick
keeps its landing record after leaveLandingInShadow, so a later throw
still resets the stored stage.
M4, a full withdrawal list. storeWithdrawal never drops an entry: a
refusal the list cannot keep pauses the base with reason `error`
(pauseForUnrecordedWithdrawals), the commit named in full in a detail
that starts "land-status withdrawals not recorded"; the stored entries
stay and are still retried while paused. The failure path does the same.
A paused base holds no blessed batch: a paused base that does not process
its batch sends it back to testing (a base-repair batch blessed under a
base-red pause whose lane no longer runs could stay at the landing stage).
A batch that did land: recordEarlierLanding, first step of every tick
(every mode, merge-ff), records a batch at the landing stage whose
candidate is the base tip as landed (afterLanded) before a pause or a
shadow tick can withdraw it; afterLanded drops stored withdrawals of the
landed commits.
Wording: the withdrawal reason is "left the landing stage" (true of a
held batch too); the repair-not-fixing comment says anyone who can label
the PR can remove the lane label.
Runbook (docs/ops/OMNI_QUEUE.md): Who lands (the two known gaps replaced
by the behaviour), State, Events (`error` row), Configuration keys
(`lanes.baseRepairLabel`), Before a mode change or a stop (steps 1 and
2), Incidents (Pause and resume; new "Paused: land-status withdrawals
not recorded").
Tests re-aimed to the new wording only ("batch abandoned" -> "left the
landing stage", assertions unchanged otherwise): "live: a batch stored at
the landing stage whose base moved is not blessed again; its success is
withdrawn", "a withdrawal refused twice in one tick is stored and retried
by the next tick", "a later status by another user does not hide the
App's success: a stored withdrawal writes failure, kept until it
succeeds", "a later status by another user does not hide the App's
success from an abandoned landing-stage batch", "a success of unknown
origin counts as the App's: an abandoned landing-stage batch writes
failure", "a stored withdrawal of a success of unknown origin writes
failure, kept until the write succeeds", "a newer status of unknown
origin never hides the App's success from a stored withdrawal", "a newer
status of unknown origin never hides the App's success from a paused
landing-stage batch", "stored withdrawals are retried in shadow mode too
(a queue switched from live), never in a dry-run", "live -> shadow: a
batch a live tick blessed and left at the landing stage loses its status
on the first shadow tick", "squash-api: a stored withdrawal of a head
this tick blesses never un-blesses it", the release-freeze test with a
blessed batch, the storedWithdrawal fixture and state.test.mjs's
withdrawalEntry fixture. "a repair taken on an unpaused broken base ..."
now checks the new comment text (and that it no longer says an approver
must remove the label).
Deleted, with what replaces them:
- "at most 20 stored withdrawals, one per SHA: when full the oldest is
dropped with an error event" (the ruled behaviour changed): replaced by
"M4: twenty stored withdrawals and a twenty-first refusal: the base
pauses, nothing is dropped, the detail names the SHA", which keeps its
checks of the 20 entries and their attempts and asserts the pause and
the absence of the drop.
- "a state without landWithdrawals loads" (cannot fail: the fixture never
had the field): the load is checked by state.test.mjs "load: a state
without landWithdrawals loads (a document of the older code)", and "no
field when nothing is refused" by the new M1 probe and the
landed-batch tests.
New tests (all failed on 05b9bd4 except the two marked): the M1 probe
(six throwing ticks), a throw before any blessing writes nothing (passed
at once), a read failing before any decision, a dry-run that throws
(passed at once), a refused withdrawal of a throwing landing, the three
M4 tests, the main()-level M1 test in shadow mode, a landed batch met by
a manual pause and by a shadow tick, a stored withdrawal of a landed
candidate, the base-red pause whose repair lane no longer runs.
This commit is contained in:
@@ -177,7 +177,17 @@ withdrawal only ever writes `failure`, so this side is the safe one). A batch th
|
||||
that stage instead of landed goes back to the stage `testing`, so the same tick withdraws its
|
||||
status too: a manual pause, a release-freeze that holds the green
|
||||
batch, or checks that went back to pending. The batch is blessed again only right before it lands
|
||||
(after the resume, when the freeze lifts, when the checks are green again).
|
||||
(after the resume, when the freeze lifts, when the checks are green again). A paused base whose
|
||||
tick does not process the batch in flight sends it back to `testing` the same way, whatever the
|
||||
pause reason (a base-repair batch blessed under a `base-red` pause whose lane no longer runs, for
|
||||
example): a paused base never holds a batch at the landing stage.
|
||||
|
||||
Before any of this, every tick (every mode, `merge-ff`) first checks whether a batch stored at the
|
||||
landing stage is already the tip of the base: then it landed and its tick lost the state save. The
|
||||
tick records it as landed (`landed` event, counters, the green verdict of the new tip, and in live
|
||||
mode the labels and comments of a landing) and writes no `failure` on the tip. A pause tick or a
|
||||
shadow tick that meets such a batch records it the same way; shadow mode lands nothing, but it
|
||||
records a landing a live tick made.
|
||||
|
||||
A withdrawal GitHub refuses while the landing withdraws its own blessing (the landing ended with a
|
||||
verdict other than landed) is tried once more by the end-of-tick pass above; other refusals wait
|
||||
@@ -197,25 +207,26 @@ landing stage, so a switch from live to shadow strands no blessing (see
|
||||
[Before a mode change or a stop](#before-a-mode-change-or-a-stop)); a queue that was never live
|
||||
has nothing to withdraw.
|
||||
|
||||
A tick that fails before the landing stage is saved saves nothing but an `error` event. One that
|
||||
fails after it has already saved that stage, the attempt count and everything the tick did before
|
||||
it: a withdrawal refused later in that tick is not stored, but its batch is still stored at the
|
||||
landing stage, and the next tick either lands it or withdraws its status as above.
|
||||
A tick that throws (any error but a lost compare-and-swap, which `main()` redoes; never in a
|
||||
dry-run) saves nothing of its own. Before it fails, it withdraws the land status of the batch it
|
||||
found stored at the landing stage, unless it landed that batch, the batch's candidate is the base
|
||||
tip (it landed; the next tick records it), or the tick made its own landing attempt. Then the
|
||||
failure path of `main()` saves, in the same save as the `error` event: that batch back at the stage
|
||||
`testing`, the withdrawals GitHub refused during the tick (stored in `landWithdrawals` as above),
|
||||
and the stored withdrawals the tick settled (dropped). So a run of throwing ticks writes `failure`
|
||||
once, on the first, and the stored state never shows a landing in progress for a batch whose
|
||||
status was withdrawn (a base that moved and whose recomposed candidate cannot be pushed, tick after
|
||||
tick, ends there). A tick that throws during its own landing attempt (stored at the landing stage
|
||||
and counted just before its blessing) withdraws its own blessing before it fails, and its batch
|
||||
stays stored at the landing stage with its `landAttempts`: that count is how the attempts limit
|
||||
(below) ends a landing that fails on every tick.
|
||||
|
||||
Two cases are not closed:
|
||||
|
||||
- a tick that throws late, after it abandoned a batch at the landing stage and before its
|
||||
end-of-tick pass, leaves the status `success`, and the batch stays stored at the landing stage.
|
||||
The attempts limit (below) does not bound this: it counts only the ticks that store the landing
|
||||
stage again before a blessing. When every later tick throws before that point too (for example
|
||||
the base moved and each tick's push of the recomposed candidate fails), the stage stays
|
||||
`landing`, `landAttempts` is never set, nothing pauses and the status stays `success`, with no
|
||||
bound. A pause of the base withdraws it (`npm run queue -- pause <base>`: the pause tick sends
|
||||
the batch back to `testing` and withdraws its status). This is a known limit, to be closed
|
||||
before live mode;
|
||||
- the list holds at most 20 entries (`LAND_WITHDRAWALS_LIMIT` in
|
||||
`scripts/merge-queue/contracts.mjs`); when it is full the oldest entry is dropped with an `error`
|
||||
event that says its newest status may still be `success`.
|
||||
The list holds at most 20 entries (`LAND_WITHDRAWALS_LIMIT` in
|
||||
`scripts/merge-queue/contracts.mjs`). Nothing is ever dropped from it: when it is full and GitHub
|
||||
refuses one more withdrawal of a commit it does not hold, the base is paused with reason `error`
|
||||
and the commit is named in full in the pause detail, which starts with
|
||||
`land-status withdrawals not recorded` (see [Incidents](#incidents)). The stored entries stay and
|
||||
are still retried on every tick of the paused base.
|
||||
|
||||
In live mode:
|
||||
|
||||
@@ -489,7 +500,8 @@ shadow bookkeeping, counters and the last 200 history events. Two fields belong
|
||||
instead of blessing it again.
|
||||
- `landWithdrawals`: present only when it is not empty. Each entry is a commit whose `failure` land
|
||||
status GitHub refused to write (`sha`, `batchId`, `candidateSha`, `why`, `since`, `attempts`),
|
||||
oldest first, one per SHA, at most 20. Later ticks retry them.
|
||||
oldest first, one per SHA, at most 20. Later ticks retry them. A refusal that a full list cannot
|
||||
keep pauses the base instead of replacing an entry.
|
||||
|
||||
A document without these fields loads as before, and code that predates them loads a document that
|
||||
has them (`validateState` ignores fields it does not know), but that code does not act on them (see
|
||||
@@ -559,7 +571,7 @@ when this table and `EVENT_TYPES` differ.
|
||||
| `paused` | A manual pause (`npm run queue -- pause`, or `action=pause`). | `npm run queue -- resume <base>` when ready. |
|
||||
| `resumed` | A manual resume (`manual resume (was <reason>)`), a `base-red` pause lifted because the base tip moved (`base moved <sha12> → <sha12> after a base-red pause`), or, in live mode, a `base-red` pause lifted because a PR of the base-repair lane landed (`base repaired by #<n>`). | Nothing. |
|
||||
| `tree-divergence` | `squash-api` only: the base tree after the merges differs from the candidate tree, or an open divergence issue stops a landing. The base is paused with reason `tree-divergence`. | Compare the trees, fix the base, close the issue, resume (see [Incidents](#incidents)). |
|
||||
| `error` | A tick failed; a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused, or a stored one dropped because the list was full; a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed. The detail says which. | Read the detail: each pause detail has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
|
||||
| `error` | A tick failed; a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused; a refused withdrawal the full `landWithdrawals` list (20) cannot keep, which pauses the base (`land-status withdrawals not recorded: …`); a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed. The detail says which. | Read the detail: each pause detail has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
|
||||
| `queue-stalled` | The queue stopped making progress (`assessHealth` in `scripts/merge-queue/dashboard.mjs`): paused for more than 120 minutes, a candidate in flight for more than `checksTimeoutMinutes` + 60 (a batch held by a release freeze is waiting, not stalled; once the hold ends, counted from its end: `inFlight.freezeHold`), eligible PRs and nothing started for more than `batch.maxWaitMinutes` + 30, or three ticks in a row that failed (`failedTicks`, counted by the failure path and deleted by the next tick that completes; `error` events appended by ticks that completed do not count). Recorded once per episode, with `stalled` in the state and one comment on the dashboard issue that mentions the approvers. Until that comment is posted (`notified` in `stalled`), every tick, completed or failed, tries again; a failed tick creates the dashboard issue when there is none. The detail lists the reasons. | Read the reasons; each points at its cause (the pause, the candidate's checks, the eligible PRs, the failing tick's `error` events). |
|
||||
|
||||
## Configuration keys
|
||||
@@ -601,7 +613,7 @@ and the keys `validateConfig` reads differ.
|
||||
| `prChecksRequired` | list of distinct check names | `[]`; shipped `Gate / CI`, `Gate / Quality` | Green on a PR head to enter a batch: worst of the suites reporting it; if they disagree, a run superseded by a newer run of the same workflow and event drops ([Eligibility](#eligibility)). |
|
||||
| `protectedPaths` | list of globs, at least one | none; shipped the agent-instruction surfaces and the queue's control plane | A PR that touches one also needs `protectedApprovalLabel` (see [Eligibility](#eligibility)). |
|
||||
| `lanes.hotfixLabel` | label name, distinct from every other label of the config | no `lanes` = no lanes; shipped `omni-queue-hotfix` | Applied by an approver to an eligible PR: at the next batch start it is composed alone, ahead of the FIFO. Never replaces what is in flight. Repeated in the tick's job filter. |
|
||||
| `lanes.baseRepairLabel` | label name, distinct from every other label of the config | no `lanes` = no lanes; shipped `omni-queue-base-repair` | Applied by an approver: composed alone (PR gates not required), the only work of a `base-red` pause; its landing lifts the pause. Never in a FIFO batch. Repeated in the tick's job filter. A repair that fails like the base is not composed again at that head (`repairTried`): push a fix (a new head is tried again), or have an approver remove this label so the PR goes back to the ordinary queue. |
|
||||
| `lanes.baseRepairLabel` | label name, distinct from every other label of the config | no `lanes` = no lanes; shipped `omni-queue-base-repair` | Applied by an approver: composed alone (PR gates not required), the only work of a `base-red` pause; its landing lifts the pause. Never in a FIFO batch. Repeated in the tick's job filter. A repair that fails like the base is not composed again at that head (`repairTried`): push a fix (a new head is tried again), or remove this label (anyone who can label the PR can): the PR rejoins the FIFO. |
|
||||
|
||||
## CLI
|
||||
|
||||
@@ -728,27 +740,22 @@ queue to shadow (or shadow to live), turning the kill switch off or rolling back
|
||||
batch at the landing stage back to `testing` and withdraws its land status; a paused base's
|
||||
ticks still retry the stored withdrawals.
|
||||
|
||||
Before you pause, check that a batch at stage `landing` is not already the tip of the base:
|
||||
`npm run queue -- status --base <base>` prints `stage <stage> · candidate <sha12>` for the batch
|
||||
in flight; compare that candidate with the tip (`git ls-remote origin refs/heads/<base>`). If
|
||||
they match, that batch did land (its push went through and the state save was lost): let the
|
||||
next live tick record it before you pause or change the mode. With `merge-ff` that tick logs
|
||||
`<batchId> is already the tip of <base>` and records a `landed` event. If the pause or a shadow
|
||||
tick meets it first, it is treated as abandoned: the base tip gets an `omni-queue/land`
|
||||
`failure` status (`omni-queue <batchId>: landing withdrawn (batch abandoned)`). After a pause,
|
||||
the first live tick after the resume still finds the candidate at the tip and records `landed`;
|
||||
when the base runs in shadow instead, history records no `landed` event for that batch. Either
|
||||
way it is harmless for the branch, whose commits are already in the base, and nothing needs
|
||||
undoing.
|
||||
A batch at stage `landing` whose candidate is already the tip of the base did land (its push
|
||||
went through and the state save was lost). With `merge-ff`, the pause tick (and a shadow tick
|
||||
too) checks that first, logs
|
||||
`<batchId> is already the tip of <base>: it landed, its save was lost`, records a `landed`
|
||||
event and writes no status on the tip. A batch that did not land is sent back to `testing` and
|
||||
its status withdrawn (`omni-queue <batchId>: landing withdrawn (left the landing stage)`).
|
||||
|
||||
2. Check that no batch is at stage `landing` (`npm run queue -- status --base <base>` prints
|
||||
`stage <stage>` for the batch in flight) and that `landWithdrawals` is empty: the field is absent
|
||||
from `state/<sanitized-base>.json` on `omni-queue-state` (an empty list leaves no field). While
|
||||
entries remain, wait for the next ticks to retry them and check again. A stored withdrawal
|
||||
dropped because the list was full (20 entries) leaves no field behind either, so also look in
|
||||
the history (`npm run queue -- history --base <base> --limit <n>`) for its `error` event:
|
||||
`<batchId>: stored withdrawal of <landing.statusContext> on <sha12> dropped (20 stored already); its newest status may still be success`.
|
||||
Each one names a commit whose land status may still be `success`: check it before the change.
|
||||
entries remain, wait for the next ticks to retry them and check again. Nothing is dropped from
|
||||
that list: a refused withdrawal that a full list (20 entries) cannot keep pauses the base with
|
||||
reason `error` and a detail that starts with `land-status withdrawals not recorded` and names
|
||||
each such commit in full. If the pause detail says so, handle it first (see
|
||||
[Incidents](#incidents), "Paused: land-status withdrawals not recorded"): each commit it names
|
||||
may still carry the App's `success`.
|
||||
|
||||
3. Only then make the change. After a mode change, resume each base
|
||||
(`npm run queue -- resume <base>`).
|
||||
@@ -1308,7 +1315,8 @@ candidate is picked up again after `npm run queue -- resume <base>`. A paused ba
|
||||
run: they retry the stored land-status withdrawals (see [Who lands](#who-lands-landingactor)), take
|
||||
the PR snapshot and run eligibility (dashboard, eligibility comments and label removals), delete
|
||||
stray candidate refs and update the dashboard. The pause tick itself sends a batch at the landing
|
||||
stage back to `testing` and withdraws its land status. To stop every base, use the kill switch, and cancel any
|
||||
stage back to `testing` and withdraws its land status (with `merge-ff`, a batch whose candidate is
|
||||
already the base tip is recorded as landed instead). To stop every base, use the kill switch, and cancel any
|
||||
running `omni-queue` run (see [Kill switch](#kill-switch)).
|
||||
|
||||
**Recomposition when the base moves.** A `land-rejected` event whose detail starts with
|
||||
@@ -1481,6 +1489,31 @@ blessing. The status of the batch's commits was withdrawn. Read the last `error`
|
||||
that outlasts three ticks ends here too: a 403 that says `rate limit` is retried, not treated as a
|
||||
refused token. Once the cause is gone, `npm run queue -- resume <base>` recomposes the batch.
|
||||
|
||||
**Paused: land-status withdrawals not recorded.** The pause detail
|
||||
`land-status withdrawals not recorded: landWithdrawals already holds 20 entries, so the refused withdrawal of <landing.statusContext> on <sha> (<batchId>) is stored nowhere (nothing was dropped). …`
|
||||
means GitHub refused to write the `failure` land status on that commit while the list of stored
|
||||
withdrawals was full, so no tick will retry it. Its newest land status by the App may still be
|
||||
`success`, which the release ruleset accepts: the commit can be pushed or merged into the base by
|
||||
hand. A pause of this kind already in place gets the next such commits appended
|
||||
(`; also not recorded: …`); a pause it replaced is quoted at the end (`(was paused: …)`). Check,
|
||||
in order:
|
||||
|
||||
1. Why GitHub refuses the App's status writes: the run's log names the error of each refused
|
||||
write, and so does an `error` event (`<batchId>: could not withdraw …`, or the detail of the
|
||||
failed tick's own `error` event when the tick threw). A revoked
|
||||
`statuses: write` permission of the App, an uninstalled App or a GitHub incident are the usual
|
||||
causes.
|
||||
2. Once that is fixed, the next ticks of the paused base retry the stored withdrawals; wait until
|
||||
`landWithdrawals` is gone from `state/<sanitized-base>.json` (or has room).
|
||||
3. The land status of each commit the detail names
|
||||
(`gh api repos/<owner>/<repo>/commits/<sha>/statuses`, newest first, context
|
||||
`<landing.statusContext>`, creator `<appSlug>[bot]`). Where the App's newest is `success`, add
|
||||
an entry for that commit to `landWithdrawals` with an ordinary commit on `omni-queue-state`
|
||||
(same fields as the others; with the queue-refs ruleset applied, disable it first and enable it
|
||||
again after): the next tick writes `failure` there.
|
||||
|
||||
Then `npm run queue -- resume <base>`: the batch in flight, if any, goes on from where it was.
|
||||
|
||||
**Labels with no effect.** If labeled PRs stay `approval-missing` and no `omni-queue/approval`
|
||||
status appears on their heads, the `approval` job is not running: look for failed
|
||||
`pull_request_target` runs with an "is not allowed to trigger Actions workflows" error (see
|
||||
|
||||
@@ -33,7 +33,8 @@ export const HISTORY_LIMIT = 200;
|
||||
export const BASE_CHECK_LIMIT = 10;
|
||||
/**
|
||||
* Land-status withdrawals GitHub refused, kept for the next ticks to retry (state.landWithdrawals,
|
||||
* one per SHA). When full, the oldest entry is dropped (the tick records an `error` event).
|
||||
* one per SHA). When full, nothing is dropped: a refusal the list cannot keep pauses the base with
|
||||
* reason `error`, the commit named in the pause detail (tick pauseForUnrecordedWithdrawals).
|
||||
*/
|
||||
export const LAND_WITHDRAWALS_LIMIT = 20;
|
||||
|
||||
|
||||
@@ -30,7 +30,9 @@
|
||||
// repository; a non-http remote is fetched without a partial-clone
|
||||
// filter). The workflow never sets it.
|
||||
//
|
||||
// Tick algorithm (per base): load state (CAS) → handle action (pause/resume) → if paused:
|
||||
// Tick algorithm (per base): load state (CAS) → a batch stored at the landing stage whose
|
||||
// candidate is already the base tip is recorded as landed (merge-ff, every mode:
|
||||
// recordEarlierLanding) → handle action (pause/resume) → if paused:
|
||||
// auto-resume a base-red pause when the base tip moved → retry the stored land-status
|
||||
// withdrawals (every mode, paused or not; never in a dry-run) → in shadow, a batch stored at the
|
||||
// landing stage goes back to "testing" and its land status is withdrawn → if paused: stop →
|
||||
@@ -47,7 +49,10 @@
|
||||
// `error` event and one more failed tick in a row (state.failedTicks, deleted by the next tick
|
||||
// that completes), then the same health verdict on the stored state, which raises the same alert
|
||||
// (three failed ticks in a row, a pause too old) and retries an undelivered one, creating the
|
||||
// dashboard issue when there is none, without changing the failed tick's outcome.
|
||||
// dashboard issue when there is none, without changing the failed tick's outcome. The same save
|
||||
// records what the failed tick did before it threw (Tick.settleFailure → failedTickRecord →
|
||||
// amendFailedTick): a batch it loaded at the landing stage and did not land had its land status
|
||||
// withdrawn by the tick and goes back to "testing", and the withdrawals GitHub refused are stored.
|
||||
// With config.batch.adaptive the batch size is state.batchWindow (eligibility.currentBatchWindow):
|
||||
// halved when a root batch is red on its own PRs (onPrRed), one more when a root batch that
|
||||
// filled it lands or would land (growBatchWindow).
|
||||
@@ -77,8 +82,14 @@
|
||||
// the batch is not blessed again and the queue pauses with reason "error". A batch HELD at
|
||||
// the landing stage (release-freeze, checks back to pending, manual pause) goes back to stage
|
||||
// "testing" (leaveLandingStage), so the same tick withdraws its status and later held ticks
|
||||
// read nothing; it is blessed again only right before it lands. A `failure` GitHub refuses
|
||||
// to write is kept in state.landWithdrawals and retried by the next ticks (retryWithdrawals).
|
||||
// read nothing; it is blessed again only right before it lands. A paused base that does not
|
||||
// process its batch sends it back to "testing" too (a paused base holds no blessed batch). A
|
||||
// tick that throws withdraws the status of a batch it loaded at the landing stage before
|
||||
// rethrowing (settleFailure), unless it landed it or made its own landing attempt (that
|
||||
// attempt withdrew its own blessing and keeps its count). A `failure` GitHub refuses to write
|
||||
// is kept in state.landWithdrawals and retried by the next ticks (retryWithdrawals); when that
|
||||
// list is full (LAND_WITHDRAWALS_LIMIT) nothing is dropped: the base pauses with reason
|
||||
// "error" and the pause detail names the commit (pauseForUnrecordedWithdrawals).
|
||||
// Before landing (or composing a bisection segment) every PR is re-read and re-run through the
|
||||
// eligibility rules: block labels, approval status on the head, protected approval and an open
|
||||
// release-freeze (which holds the green batch in flight instead of landing it). Then the candidate
|
||||
@@ -514,6 +525,21 @@ class Tick {
|
||||
this.landing = null;
|
||||
/** batchId of the batch this tick landed (its land status stays). */
|
||||
this.landedBatch = null;
|
||||
/** batchId of the batch this tick stored at the landing stage itself (markLanding). */
|
||||
this.attemptedBatch = null;
|
||||
/** The landing record (this.landing) withdrawAbandoned already handled this tick. */
|
||||
this.withdrawnLanding = null;
|
||||
/** True once this tick read the base tip for the stored landing batch (recordEarlierLanding). */
|
||||
this.tipChecked = false;
|
||||
/**
|
||||
* Withdrawals GitHub refused this tick ({sha, batchId, candidateSha, why, at}, in order), and
|
||||
* SHAs whose stored withdrawal retryWithdrawals settled: a tick that throws saves neither, so
|
||||
* main()'s failure path records them (failedTickRecord, amendFailedTick).
|
||||
*/
|
||||
this.refused = [];
|
||||
this.retired = [];
|
||||
/** Refused withdrawals a full landWithdrawals list could not keep (pauseIfUnrecorded). */
|
||||
this.unrecorded = [];
|
||||
/** {ref, result} of this tick's queue-refs protection read (queueRefsProtection). */
|
||||
this.queueRefs = null;
|
||||
}
|
||||
@@ -710,7 +736,19 @@ class Tick {
|
||||
// Refs of the stored in-flight work stay on origin until the save that supersedes them
|
||||
// succeeds: if this save loses the CAS race, the redo still finds them.
|
||||
const loadedRefs = this.refsOf(state.inFlight);
|
||||
try {
|
||||
return await this.runLoaded(loadedRefs);
|
||||
} catch (error) {
|
||||
await this.settleFailure(error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/** run() once the state is loaded: everything a failing tick hands to settleFailure. */
|
||||
async runLoaded(loadedRefs) {
|
||||
// First: a batch stored at the landing stage whose candidate is already the base tip landed
|
||||
// (its tick lost the save). Recorded before anything withdraws or turns it into anything else.
|
||||
await this.recordEarlierLanding();
|
||||
this.handleAction();
|
||||
await this.maybeAutoResume();
|
||||
// Before the in-flight work: a commit this tick blesses for landing is never un-blessed by a
|
||||
@@ -723,7 +761,13 @@ class Tick {
|
||||
const f = this.s.inFlight;
|
||||
if (f && (!this.s.paused || (this.repairOnlyPause() && f.lane === "base-repair"))) {
|
||||
await this.processInFlight();
|
||||
} else if (f) this.note(`paused: in-flight ${f.batchId} left as is`);
|
||||
} else if (f) {
|
||||
// A paused base never keeps a batch at the landing stage (a base-repair batch blessed under
|
||||
// a base-red pause whose lane no longer runs, say): back to testing, its status withdrawn
|
||||
// by this tick, as a manual pause does.
|
||||
this.leaveLandingStage(f);
|
||||
this.note(`paused: in-flight ${f.batchId} left as is`);
|
||||
}
|
||||
|
||||
const queue = await this.snapshotQueue();
|
||||
if (!this.s.paused && !this.s.inFlight) await this.startBatch(queue);
|
||||
@@ -732,6 +776,7 @@ class Tick {
|
||||
} else if (this.s.paused) this.note(`paused (${this.s.paused.reason}): nothing composed`);
|
||||
|
||||
await this.withdrawAbandoned();
|
||||
await this.pauseIfUnrecorded();
|
||||
await this.cleanupRefs(loadedRefs);
|
||||
this.endFreezeHold();
|
||||
const undelivered = this.trackHealth(queue.entries);
|
||||
@@ -836,7 +881,7 @@ class Tick {
|
||||
this.note(`blessed ${head12(sha)} (${statusContext}) for ${by}: ${reason}`);
|
||||
// An approver blesses this commit on purpose: a stored withdrawal of it (retryWithdrawals)
|
||||
// would otherwise turn this blessing into failure on the next tick.
|
||||
this.setLandWithdrawals((this.s.landWithdrawals ?? []).filter((w) => w.sha !== sha));
|
||||
this.dropWithdrawals([sha]);
|
||||
await this.saveState(state, version);
|
||||
return { base: this.base, decisions: this.decisions, state: this.s };
|
||||
}
|
||||
@@ -1183,8 +1228,27 @@ class Tick {
|
||||
const f = this.s.inFlight;
|
||||
if (!this.shadow || f?.stage !== "landing") return;
|
||||
this.leaveLandingStage(f);
|
||||
// Withdrawn once: the end-of-tick pass skips it (withdrawnLanding), and a tick that throws
|
||||
// later still knows the stored batch left the landing stage (settleFailure).
|
||||
await this.withdrawAbandoned();
|
||||
this.landing = null; // withdrawn: the end-of-tick pass has nothing left to read
|
||||
}
|
||||
|
||||
/**
|
||||
* First step of every tick (every mode, merge-ff): a batch stored at the landing stage whose
|
||||
* candidate is already the base tip did land; its tick lost the state save. It is recorded as
|
||||
* landed (afterLanded: `landed` event, counters, green base verdict, labels and comments in
|
||||
* live mode) before a pause, a shadow tick or a withdrawal can treat it as abandoned and write
|
||||
* `failure` on the tip. Shadow mode lands nothing, but records a landing an earlier live tick
|
||||
* made. One tip read, only for a batch stored at the landing stage.
|
||||
*/
|
||||
async recordEarlierLanding() {
|
||||
const f = this.s.inFlight;
|
||||
if (f?.stage !== "landing" || this.config.landing.mode !== "merge-ff") return;
|
||||
const tip = await this.gh.getBranchSha(this.base);
|
||||
this.tipChecked = true;
|
||||
if (tip !== f.candidateSha) return;
|
||||
this.note(`${f.batchId} is already the tip of ${this.base}: it landed, its save was lost`);
|
||||
await this.afterLanded(f, f.candidateSha);
|
||||
}
|
||||
|
||||
async landOrWouldLand(f) {
|
||||
@@ -1480,8 +1544,10 @@ class Tick {
|
||||
// Its commits are blessed on purpose now: a stored withdrawal of one of them is void. Dropped
|
||||
// in this save, so a tick that dies after the blessing leaves the landing stage (which
|
||||
// withdrawAbandoned reads) and no stale entry that would un-bless a commit the batch lands.
|
||||
const blessing = new Set(this.landShas(f));
|
||||
this.setLandWithdrawals((this.s.landWithdrawals ?? []).filter((w) => !blessing.has(w.sha)));
|
||||
this.dropWithdrawals(this.landShas(f));
|
||||
// This tick's own attempt: a failure after it keeps the landing stage and its count (the
|
||||
// attempts bound), and its refused withdrawals are stored by main()'s failure path.
|
||||
this.attemptedBatch = f.batchId;
|
||||
const message =
|
||||
`omni-queue: landing ${f.batchId} on ${this.base}` +
|
||||
(landAttempts > 1 ? ` (attempt ${landAttempts})` : "");
|
||||
@@ -1541,8 +1607,8 @@ class Tick {
|
||||
* Write `failure` in landing.statusContext on each of `shas`, attempting every one. A refused
|
||||
* write is recorded (an `error` event and a note), stored in state.landWithdrawals for the next
|
||||
* ticks to retry (storeWithdrawal), and returned; nothing is thrown. A tick that then throws
|
||||
* saves none of it, but its batch is still stored at the landing stage (markLanding), which the
|
||||
* next tick either lands or withdraws (withdrawAbandoned).
|
||||
* does not save its state: main()'s failure path stores the refused withdrawals instead
|
||||
* (failedTickRecord, amendFailedTick).
|
||||
* @returns {Promise<string|null>}
|
||||
*/
|
||||
async withdrawLandStatus(f, shas, why) {
|
||||
@@ -1584,15 +1650,18 @@ class Tick {
|
||||
* (newestAppLandStatus) and write `failure` where it is `success`. Only a batch at the landing
|
||||
* stage pays these reads.
|
||||
* Every mode (a batch stored at the landing stage in shadow was blessed by live mode before a
|
||||
* switch; only live stores one), never in a dry-run.
|
||||
* switch; only live stores one), never in a dry-run. Once per landing record and tick.
|
||||
* `force` (settleFailure: the tick threw, so it lands nothing more) withdraws a batch still kept
|
||||
* at the landing stage in memory too.
|
||||
*/
|
||||
async withdrawAbandoned() {
|
||||
async withdrawAbandoned({ force = false } = {}) {
|
||||
const f = this.landing;
|
||||
if (!f || this.dryRun || this.landedBatch === f.batchId) return;
|
||||
if (!f || this.dryRun || this.landedBatch === f.batchId || this.withdrawnLanding === f) return;
|
||||
const now = this.s.inFlight;
|
||||
const kept =
|
||||
now?.stage === "landing" && now.batchId === f.batchId && now.candidateSha === f.candidateSha;
|
||||
if (kept) return;
|
||||
if (kept && !force) return;
|
||||
this.withdrawnLanding = f;
|
||||
const blessed = [];
|
||||
for (const sha of this.landShas(f)) {
|
||||
try {
|
||||
@@ -1603,7 +1672,85 @@ class Tick {
|
||||
blessed.push(sha); // unknown: withdraw it anyway
|
||||
}
|
||||
}
|
||||
if (blessed.length) await this.withdrawLandStatus(f, blessed, "batch abandoned");
|
||||
// True of every batch this reaches: dropped, recomposed, paused, held, or a failed tick's.
|
||||
if (blessed.length) await this.withdrawLandStatus(f, blessed, "left the landing stage");
|
||||
}
|
||||
|
||||
/**
|
||||
* run() threw `error`: everything here is best effort, never throws and never replaces the
|
||||
* error (main() records and rethrows it as it is). Skipped for a dry-run (it writes nothing)
|
||||
* and for a lost compare-and-swap (main() redoes the tick on the stored state).
|
||||
* - The batch this tick loaded at the landing stage (or stored there itself) and did not land
|
||||
* may carry the land status: a tick that throws lands nothing more, so its blessing is
|
||||
* withdrawn now (withdrawAbandoned, forced), unless its candidate is the base tip (it
|
||||
* landed; the next tick records it). Not when this tick made its own landing attempt
|
||||
* (markLanding): that attempt withdrew its own blessing before rethrowing (revokeOrNote),
|
||||
* and its landing stage with `landAttempts` is the record of the attempts bound.
|
||||
* - What the stored state must learn from this tick is handed to main()'s failure path, which
|
||||
* saves it with the `error` event (failedTickRecord, amendFailedTick): the landing stage of
|
||||
* a batch withdrawn here is left (back to testing), the withdrawals GitHub refused are
|
||||
* stored, and the stored withdrawals this tick settled are dropped.
|
||||
*/
|
||||
async settleFailure(error) {
|
||||
if (this.dryRun || error instanceof StateConflictError) return;
|
||||
let leaveLanding = null;
|
||||
try {
|
||||
const f = this.landing;
|
||||
const settled = !f || this.landedBatch === f.batchId || this.attemptedBatch === f.batchId;
|
||||
if (!settled && !(await this.isBaseTip(f))) {
|
||||
const before = this.refused.length;
|
||||
await this.withdrawAbandoned({ force: true });
|
||||
leaveLanding = { batchId: f.batchId, candidateSha: f.candidateSha };
|
||||
const refused = this.refused.slice(before);
|
||||
if (refused.length && error instanceof Error) {
|
||||
error.message +=
|
||||
`; the tick's withdrawal of ${f.batchId} was refused on ` +
|
||||
`${refused.map((w) => head12(w.sha)).join(", ")} (stored for retry)`;
|
||||
}
|
||||
}
|
||||
} catch (failure) {
|
||||
this.note(`after the failure: ${String(failure?.message ?? failure).slice(0, 200)}`);
|
||||
}
|
||||
if (!leaveLanding && !this.refused.length && !this.retired.length) return;
|
||||
if (error === null || (typeof error !== "object" && typeof error !== "function")) return;
|
||||
FAILED_TICKS.set(error, {
|
||||
leaveLanding,
|
||||
withdrawals: this.refused.map((w) => ({ ...w })),
|
||||
retired: [...this.retired],
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* merge-ff: whether `f`'s candidate is the base tip (it landed). Never throws: an unreadable tip
|
||||
* is false (the withdrawal goes ahead). Not read again when recordEarlierLanding read it.
|
||||
*/
|
||||
async isBaseTip(f) {
|
||||
if (this.config.landing.mode !== "merge-ff" || this.tipChecked) return false;
|
||||
try {
|
||||
return (await this.gh.getBranchSha(this.base)) === f.candidateSha;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* End of a tick: the withdrawals GitHub refused this tick that a full landWithdrawals list could
|
||||
* not keep (storeWithdrawal) pause the base with reason `error`, naming each commit (pure
|
||||
* pauseForUnrecordedWithdrawals). The pause leaves a landing stage (a paused base holds no
|
||||
* blessed batch), so withdrawAbandoned runs once more; a refusal there is named as well.
|
||||
*/
|
||||
async pauseIfUnrecorded() {
|
||||
for (let pass = 0; pass < 2 && this.unrecorded.length; pass++) {
|
||||
const unrecorded = this.unrecorded;
|
||||
this.unrecorded = [];
|
||||
this.s = pauseForUnrecordedWithdrawals(this.s, unrecorded, {
|
||||
at: this.now(),
|
||||
url: this.deps.runUrl ?? undefined,
|
||||
statusContext: this.config.landing.statusContext,
|
||||
});
|
||||
this.note(`paused: ${this.s.paused.detail}`);
|
||||
await this.withdrawAbandoned();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1635,35 +1782,23 @@ class Tick {
|
||||
}
|
||||
|
||||
/**
|
||||
* Keep a `failure` GitHub refused to write on `sha` for the next ticks (retryWithdrawals): one
|
||||
* entry per SHA (a new refusal updates `why` and counts an attempt, `since` stays), at most
|
||||
* LAND_WITHDRAWALS_LIMIT. A full list drops its oldest entry, and an `error` event says which.
|
||||
* Keep a `failure` GitHub refused to write on `sha` for the next ticks (retryWithdrawals), by
|
||||
* the pure storeWithdrawalIn: one entry per SHA, at most LAND_WITHDRAWALS_LIMIT. Nothing is ever
|
||||
* dropped: a refusal the full list cannot keep is held in this.unrecorded, and the end of the
|
||||
* tick pauses the base naming it (pauseIfUnrecorded). Every refusal is also kept in
|
||||
* this.refused for main()'s failure path.
|
||||
*/
|
||||
storeWithdrawal(f, sha, why) {
|
||||
const list = [...(this.s.landWithdrawals ?? [])];
|
||||
const i = list.findIndex((w) => w.sha === sha);
|
||||
if (i >= 0) {
|
||||
list[i] = { ...list[i], why, attempts: list[i].attempts + 1 };
|
||||
return this.setLandWithdrawals(list);
|
||||
}
|
||||
if (list.length >= LAND_WITHDRAWALS_LIMIT) {
|
||||
const dropped = list.shift();
|
||||
const detail =
|
||||
`${dropped.batchId}: stored withdrawal of ${this.config.landing.statusContext} on ` +
|
||||
`${head12(dropped.sha)} dropped (${LAND_WITHDRAWALS_LIMIT} stored already); its newest ` +
|
||||
"status may still be success";
|
||||
this.event("error", { batchId: dropped.batchId, sha: dropped.sha, detail });
|
||||
this.note(detail);
|
||||
}
|
||||
list.push({
|
||||
sha,
|
||||
batchId: f.batchId,
|
||||
candidateSha: f.candidateSha,
|
||||
why,
|
||||
since: this.now(),
|
||||
attempts: 1,
|
||||
});
|
||||
this.setLandWithdrawals(list);
|
||||
const refusal = { sha, batchId: f.batchId, candidateSha: f.candidateSha, why, at: this.now() };
|
||||
this.refused.push(refusal);
|
||||
const { state, stored } = storeWithdrawalIn(this.s, refusal);
|
||||
this.setLandWithdrawals(state.landWithdrawals ?? []);
|
||||
if (stored) return;
|
||||
this.unrecorded.push(refusal);
|
||||
this.note(
|
||||
`${f.batchId}: withdrawal of ${head12(sha)} not stored: landWithdrawals holds ` +
|
||||
`${LAND_WITHDRAWALS_LIMIT} entries; the base pauses at the end of the tick`
|
||||
);
|
||||
}
|
||||
|
||||
/** state.landWithdrawals = `list`; an empty list leaves no field (documents stay as before). */
|
||||
@@ -1672,6 +1807,15 @@ class Tick {
|
||||
else delete this.s.landWithdrawals;
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored withdrawals of `shas` go: those commits are blessed on purpose (markLanding, a
|
||||
* manual bless) or landed (afterLanded), so withdrawing them would be wrong.
|
||||
*/
|
||||
dropWithdrawals(shas) {
|
||||
const drop = new Set(shas);
|
||||
this.setLandWithdrawals((this.s.landWithdrawals ?? []).filter((w) => !drop.has(w.sha)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Retry each stored withdrawal (storeWithdrawal): read the land status that decides a
|
||||
* withdrawal on its commit (newestAppLandStatus); `success` → write `failure` again. The entry
|
||||
@@ -1707,6 +1851,7 @@ class Tick {
|
||||
`${w.batchId}: stored withdrawal of ${head12(w.sha)} dropped ` +
|
||||
`(the App's ${statusContext} is ${newest?.state ?? "absent"})`
|
||||
);
|
||||
this.retired.push(w.sha);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
@@ -1720,6 +1865,7 @@ class Tick {
|
||||
`${w.batchId}: ${statusContext} withdrawn on ${head12(w.sha)} (${w.why}; ` +
|
||||
`${w.attempts} failed attempt(s) since ${w.since})`
|
||||
);
|
||||
this.retired.push(w.sha);
|
||||
} catch (error) {
|
||||
kept.push({ ...w, attempts: w.attempts + 1 });
|
||||
this.note(
|
||||
@@ -1734,6 +1880,9 @@ class Tick {
|
||||
async afterLanded(f, sha) {
|
||||
const { label, protectedApprovalLabel } = this.config;
|
||||
this.landedBatch = f.batchId; // its land status stays: the commits are on the base
|
||||
// A stored withdrawal of a landed commit (its landing threw after the push went through and
|
||||
// the withdrawal was refused) must not write `failure` on the base.
|
||||
this.dropWithdrawals(this.landShas(f));
|
||||
// The new tip is the candidate (merge-ff) or a squash of the same tree (squash-api, checked by
|
||||
// tree-divergence), green on the profile that just ran: a red batch on it blames its PRs
|
||||
// without a base check. Recorded like any verdict, it replaces an older one for this SHA.
|
||||
@@ -2472,8 +2621,9 @@ class Tick {
|
||||
`🔴 **omni-queue**: this PR is in the base-repair lane of \`${this.base}\`, but its`,
|
||||
`candidate failed with the same failures as the base, so it does not repair it. This`,
|
||||
"head is not composed again. Two ways out: push a fix (a new head is tried again), or",
|
||||
`have an approver remove the \`${this.config.lanes.baseRepairLabel}\` label so the PR`,
|
||||
"goes back to the ordinary queue.",
|
||||
`remove the \`${this.config.lanes.baseRepairLabel}\` label: the PR goes back to the ordinary queue.`,
|
||||
"Anyone who can label this PR can remove it; it counts again only when an approver",
|
||||
"applies it.",
|
||||
"",
|
||||
`- Head: \`${pr.head}\``,
|
||||
`- Candidate: ${this.checksUrl(batch.candidateSha)} (batch \`${batch.batchId}\`)`,
|
||||
@@ -3272,6 +3422,11 @@ export function trackStall(state, health, { at, url, clear = true }) {
|
||||
* comment is not delivered yet (opened now, or by an earlier tick whose comment failed),
|
||||
* deliverStalledAlert tries again. This is what alerts a queue whose every tick throws (its
|
||||
* dashboard is never refreshed).
|
||||
* The same save applies `record`, what the failed tick left for the stored state
|
||||
* (failedTickRecord: a batch it withdrew leaves the landing stage, refused withdrawals are stored,
|
||||
* settled ones dropped; amendFailedTick), so the stored state never claims a landing in progress
|
||||
* for a batch whose status the tick withdrew. A record that cannot be applied goes to `warn` and
|
||||
* the error is recorded without it.
|
||||
* Throws only what the error record's own save throws (as before this alert existed). The
|
||||
* health verdict and the delivery never throw: a failure there goes to `warn` and is ignored, so
|
||||
* the failed tick stays failed with its own error and exit code.
|
||||
@@ -3279,13 +3434,21 @@ export function trackStall(state, health, { at, url, clear = true }) {
|
||||
* @param {string} base
|
||||
* @param {string} detail the tick's (redacted) error message
|
||||
* @param {(message: string) => void} [warn]
|
||||
* @param {ReturnType<typeof failedTickRecord>} [record]
|
||||
*/
|
||||
export async function recordTickFailure(deps, base, detail, warn = () => {}) {
|
||||
export async function recordTickFailure(deps, base, detail, warn = () => {}, record = null) {
|
||||
const saved = await updateStateWithRetry(deps.store, base, (state) => {
|
||||
const at = deps.now();
|
||||
const url = deps.runUrl ?? undefined;
|
||||
let amended = state;
|
||||
try {
|
||||
const statusContext = deps.config.landing.statusContext;
|
||||
amended = amendFailedTick(state, record, { at, url, statusContext });
|
||||
} catch (error) {
|
||||
warn(`the failed tick's record was not applied: ${error.message}`);
|
||||
}
|
||||
const failed = countFailedTick(
|
||||
appendEvent(state, stripUndefined({ at, type: "error", detail, url })),
|
||||
appendEvent(amended, stripUndefined({ at, type: "error", detail, url })),
|
||||
at
|
||||
);
|
||||
try {
|
||||
@@ -3325,6 +3488,166 @@ export function countFailedTick(state, at) {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* What a tick that threw leaves for main()'s failure path to save (Tick.settleFailure): keyed by
|
||||
* the thrown error, so the error itself is passed on untouched. See failedTickRecord.
|
||||
*/
|
||||
const FAILED_TICKS = new WeakMap();
|
||||
|
||||
/**
|
||||
* The record a tick that threw `error` left for the stored state, or null (nothing to add):
|
||||
* - leaveLanding {batchId, candidateSha}: the tick withdrew the land status of that batch,
|
||||
* stored at the landing stage, without landing it: the stored batch goes back to testing;
|
||||
* - withdrawals [{sha, batchId, candidateSha, why, at}]: `failure` writes GitHub refused;
|
||||
* - retired [sha]: stored withdrawals the tick settled (written, or nothing left to withdraw).
|
||||
* main() hands it to recordTickFailure, which applies it (amendFailedTick) in the save that
|
||||
* records the `error` event.
|
||||
* @param {unknown} error
|
||||
*/
|
||||
export function failedTickRecord(error) {
|
||||
if (error === null || (typeof error !== "object" && typeof error !== "function")) return null;
|
||||
return FAILED_TICKS.get(error) ?? null;
|
||||
}
|
||||
|
||||
/** `state` with landWithdrawals = `list` (pure); an empty list leaves no field. */
|
||||
function withLandWithdrawals(state, list) {
|
||||
const next = { ...state };
|
||||
if (list.length) next.landWithdrawals = list;
|
||||
else delete next.landWithdrawals;
|
||||
return next;
|
||||
}
|
||||
|
||||
/**
|
||||
* Store one refused withdrawal in state.landWithdrawals (pure): one entry per SHA (a new refusal
|
||||
* of a stored SHA updates `why` and counts an attempt, `since` stays), at most
|
||||
* LAND_WITHDRAWALS_LIMIT. Nothing is ever dropped to make room: when the list is full and holds
|
||||
* no entry for the SHA, `stored` is false and the state is returned unchanged (the caller pauses
|
||||
* the base, pauseForUnrecordedWithdrawals).
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string, candidateSha: string, why: string, at: string}} refusal
|
||||
* @returns {{state: QueueState, stored: boolean}}
|
||||
*/
|
||||
export function storeWithdrawalIn(state, refusal) {
|
||||
const list = [...(state.landWithdrawals ?? [])];
|
||||
const i = list.findIndex((w) => w.sha === refusal.sha);
|
||||
if (i >= 0) {
|
||||
list[i] = { ...list[i], why: refusal.why, attempts: list[i].attempts + 1 };
|
||||
return { state: withLandWithdrawals(state, list), stored: true };
|
||||
}
|
||||
if (list.length >= LAND_WITHDRAWALS_LIMIT) return { state, stored: false };
|
||||
const { sha, batchId, candidateSha, why, at } = refusal;
|
||||
list.push({ sha, batchId, candidateSha, why, since: at, attempts: 1 });
|
||||
return { state: withLandWithdrawals(state, list), stored: true };
|
||||
}
|
||||
|
||||
/** Start of the pause detail of withdrawals a full landWithdrawals list could not keep. */
|
||||
export const WITHDRAWALS_FULL_DETAIL = "land-status withdrawals not recorded";
|
||||
|
||||
/**
|
||||
* `state` with its in-flight batch at stage "testing" when it was at "landing" (pure): the
|
||||
* landing count goes with the stage, every other field stays (as Tick.leaveLandingStage).
|
||||
*/
|
||||
function leaveLandingIn(state) {
|
||||
const f = state.inFlight;
|
||||
if (f?.stage !== "landing") return state;
|
||||
const rest = { ...f };
|
||||
delete rest.landAttempts;
|
||||
return { ...state, inFlight: { ...rest, stage: "testing" } };
|
||||
}
|
||||
|
||||
/**
|
||||
* Pause the base with reason `error` for refused withdrawals that the full landWithdrawals list
|
||||
* could not keep (pure). Nothing is dropped from the list; each commit is named in full in the
|
||||
* pause detail, which says what to check, and one `error` event carries the same detail. A pause
|
||||
* of this kind already in place gets the new commits appended; any other pause is replaced and
|
||||
* quoted at the end of the detail (a base-red pause would lift by itself when the tip moves and
|
||||
* take the names with it). A batch at the landing stage goes back to testing: a paused base holds
|
||||
* no blessed batch.
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string}[]} unrecorded
|
||||
* @param {{at: string, url?: string, statusContext: string}} options
|
||||
*/
|
||||
export function pauseForUnrecordedWithdrawals(state, unrecorded, { at, url, statusContext }) {
|
||||
const paused = state.paused;
|
||||
const listed =
|
||||
paused?.reason === "error" && String(paused.detail ?? "").startsWith(WITHDRAWALS_FULL_DETAIL);
|
||||
const named = new Set();
|
||||
const fresh = [];
|
||||
for (const u of unrecorded) {
|
||||
if (named.has(u.sha) || (listed && paused.detail.includes(u.sha))) continue;
|
||||
named.add(u.sha);
|
||||
fresh.push(u);
|
||||
}
|
||||
if (!fresh.length) return leaveLandingIn(state);
|
||||
const commits = fresh.map((u) => `${u.sha} (${u.batchId})`).join(", ");
|
||||
const detail = listed
|
||||
? `${paused.detail}; also not recorded: ${commits}`
|
||||
: `${WITHDRAWALS_FULL_DETAIL}: landWithdrawals already holds ${LAND_WITHDRAWALS_LIMIT} ` +
|
||||
`entries, so the refused withdrawal of ${statusContext} on ${commits} is stored nowhere ` +
|
||||
"(nothing was dropped). Check why GitHub refuses the App's status writes, and the land " +
|
||||
"status of each commit named here (docs/ops/OMNI_QUEUE.md, Incidents); resume once " +
|
||||
"landWithdrawals has room" +
|
||||
(paused ? ` (was paused: ${paused.reason}: ${String(paused.detail).slice(0, 200)})` : "");
|
||||
const next = {
|
||||
...state,
|
||||
paused: {
|
||||
reason: "error",
|
||||
since: listed ? paused.since : at,
|
||||
baseSha: null,
|
||||
detail,
|
||||
issue: null,
|
||||
},
|
||||
};
|
||||
const event = stripUndefined({
|
||||
at,
|
||||
type: "error",
|
||||
batchId: fresh[0].batchId,
|
||||
sha: fresh[0].sha,
|
||||
detail: detail.slice(0, 500),
|
||||
url,
|
||||
});
|
||||
return leaveLandingIn(appendEvent(next, event));
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply to the stored state what a tick that threw left for it (failedTickRecord), pure: drop
|
||||
* the stored withdrawals it settled, take the batch it withdrew off the landing stage when the
|
||||
* stored batch is still that one at that stage, store the withdrawals GitHub refused (and pause
|
||||
* the base for any the full list cannot keep, pauseForUnrecordedWithdrawals). A null record
|
||||
* returns `state` unchanged.
|
||||
* @param {QueueState} state
|
||||
* @param {ReturnType<typeof failedTickRecord>} record
|
||||
* @param {{at: string, url?: string, statusContext: string}} options
|
||||
*/
|
||||
export function amendFailedTick(state, record, options) {
|
||||
if (!record) return state;
|
||||
let next = state;
|
||||
const retired = new Set(record.retired ?? []);
|
||||
if (retired.size && next.landWithdrawals) {
|
||||
next = withLandWithdrawals(
|
||||
next,
|
||||
next.landWithdrawals.filter((w) => !retired.has(w.sha))
|
||||
);
|
||||
}
|
||||
const leave = record.leaveLanding;
|
||||
const f = next.inFlight;
|
||||
if (
|
||||
leave &&
|
||||
f?.stage === "landing" &&
|
||||
f.batchId === leave.batchId &&
|
||||
f.candidateSha === leave.candidateSha
|
||||
) {
|
||||
next = leaveLandingIn(next);
|
||||
}
|
||||
const unrecorded = [];
|
||||
for (const refusal of record.withdrawals ?? []) {
|
||||
const result = storeWithdrawalIn(next, refusal);
|
||||
next = result.state;
|
||||
if (!result.stored) unrecorded.push(refusal);
|
||||
}
|
||||
return unrecorded.length ? pauseForUnrecordedWithdrawals(next, unrecorded, options) : next;
|
||||
}
|
||||
|
||||
/**
|
||||
* main()'s failure path: post the stored episode's comment on the dashboard issue and record
|
||||
* `notified: true`. A base with no dashboard issue (no tick of it has completed yet) gets one,
|
||||
@@ -3689,8 +4012,12 @@ export async function main(argv = process.argv.slice(2), env = process.env) {
|
||||
console.error(`omni-queue ${base}: ${redact(error.stack || error.message)}`);
|
||||
if (args.action === "dry-run") continue;
|
||||
try {
|
||||
await recordTickFailure(deps, base, redact(error.message).slice(0, 500), (message) =>
|
||||
console.error(`omni-queue ${base}: ${redact(message)}`)
|
||||
await recordTickFailure(
|
||||
deps,
|
||||
base,
|
||||
redact(error.message).slice(0, 500),
|
||||
(message) => console.error(`omni-queue ${base}: ${redact(message)}`),
|
||||
failedTickRecord(error)
|
||||
);
|
||||
} catch (recordError) {
|
||||
console.error(
|
||||
|
||||
@@ -297,7 +297,7 @@ function withdrawalEntry(i, extra = {}) {
|
||||
sha: String(i % 10).repeat(39) + "c",
|
||||
batchId: `b${i}`,
|
||||
candidateSha: sha("d"),
|
||||
why: "batch abandoned",
|
||||
why: "left the landing stage",
|
||||
since: "2026-11-13T12:00:00.000Z",
|
||||
attempts: 1,
|
||||
...extra,
|
||||
|
||||
@@ -36,6 +36,7 @@ import {
|
||||
TICK_ACTIONS,
|
||||
createTickDeps,
|
||||
discoverBases,
|
||||
failedTickRecord,
|
||||
hangSentence,
|
||||
main,
|
||||
parseArgs,
|
||||
@@ -2088,7 +2089,7 @@ test("live: a batch stored at the landing stage whose base moved is not blessed
|
||||
const state = await h.state();
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })],
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })],
|
||||
"not blessed again; the previous tick's success is turned into failure"
|
||||
);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
@@ -2415,6 +2416,11 @@ test("a landing failure other than base-moved/divergence pauses with reason erro
|
||||
const LAND_TOKEN_MISSING = 'landing needs OMNI_QUEUE_LAND_TOKEN (landing.actor is "owner")';
|
||||
const checksUrlOf = (sha) => `https://github.com/acme/widgets/commit/${sha}/checks`;
|
||||
const statusCalls = (h) => h.calls.filter((c) => c.name === "createCommitStatus");
|
||||
/**
|
||||
* Why a batch that left the landing stage without landing (dropped, recomposed, paused, held, or
|
||||
* a failed tick's) has its land status withdrawn: withdrawAbandoned's one string.
|
||||
*/
|
||||
const LEFT_LANDING = "left the landing stage";
|
||||
/** createCommitStatus args of the land status; the link is the candidate's checks. */
|
||||
const landStatus = (batchId, sha, { why = null, candidateSha = sha } = {}) => [
|
||||
sha,
|
||||
@@ -3100,7 +3106,7 @@ const storedWithdrawal = (sha, extra = {}) => ({
|
||||
sha,
|
||||
batchId: "b7",
|
||||
candidateSha: sha,
|
||||
why: "batch abandoned",
|
||||
why: LEFT_LANDING,
|
||||
since: "2026-11-13T11:30:00.000Z",
|
||||
attempts: 1,
|
||||
...extra,
|
||||
@@ -3231,7 +3237,7 @@ test("a withdrawal refused twice in one tick is stored and retried by the next t
|
||||
sha: f.candidateSha,
|
||||
batchId: f.batchId,
|
||||
candidateSha: f.candidateSha,
|
||||
why: "batch abandoned",
|
||||
why: LEFT_LANDING,
|
||||
since: tickAt,
|
||||
attempts: 2,
|
||||
},
|
||||
@@ -3247,7 +3253,7 @@ test("a withdrawal refused twice in one tick is stored and retried by the next t
|
||||
statusCalls(h)
|
||||
.slice(written)
|
||||
.map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })]
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })]
|
||||
);
|
||||
assert.equal("landWithdrawals" in state, false, "the marker is gone");
|
||||
assert.equal(couldNotWithdraw(state).length, 2, "a successful retry is a note, not an event");
|
||||
@@ -3301,7 +3307,7 @@ test("a later status by another user does not hide the App's success: a stored w
|
||||
state = await h.state();
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus("b7", sha, { why: "batch abandoned" })]
|
||||
[landStatus("b7", sha, { why: LEFT_LANDING })]
|
||||
);
|
||||
assert.equal(newestAppLand(h, sha), "failure", "the App's status no longer lets it in");
|
||||
assert.equal("landWithdrawals" in state, false, "dropped once the write succeeded");
|
||||
@@ -3315,7 +3321,7 @@ test("a later status by another user does not hide the App's success from an aba
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })]
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })]
|
||||
);
|
||||
assert.equal(newestAppLand(h, f.candidateSha), "failure");
|
||||
});
|
||||
@@ -3351,7 +3357,7 @@ test("a success of unknown origin counts as the App's: an abandoned landing-stag
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })],
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })],
|
||||
`creator ${field ? "null" : "absent"}: a withdrawal only ever writes failure`
|
||||
);
|
||||
}
|
||||
@@ -3389,8 +3395,8 @@ test("a stored withdrawal of a success of unknown origin writes failure, kept un
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[
|
||||
landStatus("b7", unknown, { why: "batch abandoned" }),
|
||||
landStatus("b7", absent, { why: "batch abandoned" }),
|
||||
landStatus("b7", unknown, { why: LEFT_LANDING }),
|
||||
landStatus("b7", absent, { why: LEFT_LANDING }),
|
||||
]
|
||||
);
|
||||
assert.equal(newestAppLand(h, unknown), "failure");
|
||||
@@ -3430,8 +3436,8 @@ test("a newer status of unknown origin never hides the App's success from a stor
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[
|
||||
landStatus("b7", pending, { why: "batch abandoned" }),
|
||||
landStatus("b7", errored, { why: "batch abandoned" }),
|
||||
landStatus("b7", pending, { why: LEFT_LANDING }),
|
||||
landStatus("b7", errored, { why: LEFT_LANDING }),
|
||||
]
|
||||
);
|
||||
assert.equal(newestAppLand(h, pending), "failure");
|
||||
@@ -3448,7 +3454,7 @@ test("a newer status of unknown origin never hides the App's success from a paus
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })],
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })],
|
||||
`a newer ${later} of unknown origin`
|
||||
);
|
||||
assert.equal(newestAppLand(h, f.candidateSha), "failure");
|
||||
@@ -3491,17 +3497,6 @@ test("dry-run in shadow on a batch stored at the landing stage writes nothing an
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha);
|
||||
});
|
||||
|
||||
test("a state without landWithdrawals loads", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await greenBatch(h, [1]);
|
||||
const stored = await h.state();
|
||||
assert.equal("landWithdrawals" in stored, false, "a document of the code before the field");
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.ok(types(state).includes("landed"));
|
||||
assert.equal("landWithdrawals" in state, false, "nothing refused, nothing stored");
|
||||
});
|
||||
|
||||
test("stored withdrawals are retried on a paused queue; a refusal and an unreadable status each count an attempt", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
@@ -3557,7 +3552,7 @@ test("stored withdrawals are retried in shadow mode too (a queue switched from l
|
||||
await shadow.tick();
|
||||
assert.deepEqual(
|
||||
statusCalls(shadow).map((c) => c.args),
|
||||
[landStatus("b7", sha, { why: "batch abandoned" })],
|
||||
[landStatus("b7", sha, { why: LEFT_LANDING })],
|
||||
"the stray success left by live mode is withdrawn"
|
||||
);
|
||||
assert.equal(landStatusReads(shadow, sha), 1);
|
||||
@@ -3585,7 +3580,7 @@ test("live → shadow: a batch a live tick blessed and left at the landing stage
|
||||
assert.equal(state.inFlight, null, "and clears the batch");
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })],
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })],
|
||||
"the blessing is withdrawn, not stranded"
|
||||
);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
@@ -3630,37 +3625,6 @@ test("a queue that was never live reads and writes no land status in shadow mode
|
||||
);
|
||||
});
|
||||
|
||||
test("at most 20 stored withdrawals, one per SHA: when full the oldest is dropped with an error event", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
for (const sha of old) setLand(h, sha, "success");
|
||||
await storeWithdrawals(
|
||||
h,
|
||||
old.map((sha) => storedWithdrawal(sha))
|
||||
);
|
||||
h.world.baseMovesOnLand = fakeSha("manual-merge"); // the landing is refused: base-moved
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(state.landWithdrawals.length, 20);
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((e) => e.sha),
|
||||
[...old.slice(1), f.candidateSha],
|
||||
"the oldest entry made room; the new SHA appears once"
|
||||
);
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((e) => e.attempts),
|
||||
[...Array(19).fill(2), 2],
|
||||
"each refused again (retry; path + end-of-tick pass for the new one)"
|
||||
);
|
||||
const dropped = state.history.find(
|
||||
(e) => e.type === "error" && e.sha === old[0] && /dropped/.test(e.detail ?? "")
|
||||
);
|
||||
assert.ok(dropped, "the dropped entry is named in an error event");
|
||||
assert.match(dropped.detail, new RegExp(head12(old[0])));
|
||||
});
|
||||
|
||||
for (const refused of [false, true]) {
|
||||
test(`squash-api: a stored withdrawal of a head this tick blesses never un-blesses it (retry ${refused ? "refused" : "written"})`, async () => {
|
||||
const h = createHarness({ mode: "live", landing: "squash-api", actor: "owner" });
|
||||
@@ -3680,7 +3644,7 @@ for (const refused of [false, true]) {
|
||||
[
|
||||
...(refused
|
||||
? []
|
||||
: [landStatus("b0", head, { why: "batch abandoned", candidateSha: fakeSha("b0") })]),
|
||||
: [landStatus("b0", head, { why: LEFT_LANDING, candidateSha: fakeSha("b0") })]),
|
||||
landStatus(f.batchId, head, { candidateSha: f.candidateSha }),
|
||||
],
|
||||
"the stale blessing is withdrawn first, then this batch blesses the head"
|
||||
@@ -4039,9 +4003,7 @@ for (const blessed of [false, true]) {
|
||||
// Held = not at the landing stage: a blessing is withdrawn, and the landing count goes too.
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
const withdrawn = blessed
|
||||
? [landStatus(f.batchId, f.candidateSha, { why: "batch abandoned" })]
|
||||
: [];
|
||||
const withdrawn = blessed ? [landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })] : [];
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
withdrawn,
|
||||
@@ -6334,11 +6296,16 @@ test("a repair taken on an unpaused broken base goes through a base check that c
|
||||
assert.equal(comments.length, 1);
|
||||
// both ways out of a tried repair
|
||||
assert.match(comments[0].body, /push a fix \(a new head is tried again\)/);
|
||||
assert.match(
|
||||
comments[0].body,
|
||||
new RegExp(`have an approver remove the \`${baseRepairLabel}\` label`)
|
||||
assert.ok(
|
||||
comments[0].body.includes(
|
||||
`remove the \`${baseRepairLabel}\` label: the PR goes back to the ordinary queue.`
|
||||
),
|
||||
comments[0].body
|
||||
);
|
||||
assert.match(comments[0].body, /goes back to the ordinary queue/);
|
||||
// whoever can label the PR can remove the label: no approver is needed for that
|
||||
assert.match(comments[0].body, /Anyone who can label this PR can remove it;/);
|
||||
assert.match(comments[0].body, /it counts again only when an approver\napplies it\./);
|
||||
assert.doesNotMatch(comments[0].body, /approver remove/);
|
||||
assert.ok(types(state).includes("base-red-paused"));
|
||||
assert.deepEqual([state.paused.reason, state.paused.baseSha], ["base-red", BASE0]);
|
||||
assert.ok(!types(state).includes("culprit-found"));
|
||||
@@ -7212,3 +7179,366 @@ test("a base the queue does not manage is refused before anything is loaded or w
|
||||
assert.equal(await main(["--base", q.base], q.env), 0);
|
||||
assert.ok(q.fake.readFile(STATE_BRANCH, stateFilePath(q.base)));
|
||||
});
|
||||
|
||||
// ─────────── Task 13b: withdrawal gaps closed before live mode (M1, M4, pauses, landed) ───────────
|
||||
|
||||
/**
|
||||
* One tick that throws, recorded the way main() records it: recordTickFailure with the tick's own
|
||||
* record of what it did before throwing (failedTickRecord). Returns the error.
|
||||
*/
|
||||
async function recordedFailingTick(h, pattern, options = {}) {
|
||||
let thrown = null;
|
||||
await assert.rejects(h.tick(options), (error) => {
|
||||
thrown = error;
|
||||
return pattern.test(error.message);
|
||||
});
|
||||
const record = failedTickRecord(thrown);
|
||||
await recordTickFailure(h.deps, BASE, thrown.message.slice(0, 500), () => {}, record);
|
||||
h.clock.advance(10);
|
||||
return thrown;
|
||||
}
|
||||
|
||||
/** Every candidate push of the fake origin throws (a push that gives no verdict). */
|
||||
function failPushes(h) {
|
||||
h.modules.compose.pushCandidate = () => {
|
||||
throw new Error("push: the remote end hung up unexpectedly");
|
||||
};
|
||||
}
|
||||
|
||||
test("M1 probe: a landing-stage batch whose base moved and whose recomposition push throws six times: withdrawn by the first tick, not stored as landing after it", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]); // blessed by a tick that died
|
||||
h.gh.branches.set(BASE, fakeSha("manual-merge")); // the base moved: the batch is recomposed
|
||||
failPushes(h);
|
||||
for (let tick = 1; tick <= 6; tick++) {
|
||||
await recordedFailingTick(h, /remote end hung up/);
|
||||
const state = await h.state();
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })],
|
||||
`tick ${tick}: withdrawn once, by the first failing tick`
|
||||
);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha, "the batch stays in flight");
|
||||
assert.equal(state.inFlight.stage, "testing", `tick ${tick}: no landing claimed`);
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.failedTicks, tick);
|
||||
assert.equal(
|
||||
state.history.findLast((e) => e.type === "error").detail.includes("hung up"),
|
||||
true
|
||||
);
|
||||
assert.equal("landWithdrawals" in state, false, "nothing refused, nothing stored");
|
||||
}
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
});
|
||||
|
||||
test("M1: a tick that throws before anything was blessed writes no status and records no landing", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]); // never blessed: stage testing
|
||||
h.gh.branches.set(BASE, fakeSha("manual-merge"));
|
||||
failPushes(h);
|
||||
const thrown = await recordedFailingTick(h, /remote end hung up/);
|
||||
const state = await h.state();
|
||||
assert.deepEqual(statusCalls(h), [], "no land status read or written");
|
||||
assert.equal(landStatusReads(h, f.candidateSha), 0);
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.equal("landWithdrawals" in state, false);
|
||||
assert.equal(failedTickRecord(thrown), null, "nothing to add to the record");
|
||||
});
|
||||
|
||||
test("M1: a read that fails before the tick decided anything still withdraws a stored blessing", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]);
|
||||
h.gh.getPull = async () => {
|
||||
throw new Error("GET /pulls/1 → 502: Bad Gateway");
|
||||
};
|
||||
await recordedFailingTick(h, /502/);
|
||||
const state = await h.state();
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })]
|
||||
);
|
||||
assert.equal(state.inFlight.stage, "testing", "saved with the error event");
|
||||
assert.equal(state.history.at(-1).type, "error");
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
});
|
||||
|
||||
test("M1: a dry-run that throws writes nothing, withdrawal included", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]);
|
||||
const stored = h.gh.contents.get(`omni-queue-state:${stateFilePath(BASE)}`)?.content;
|
||||
const writes = h.gh.writes.length;
|
||||
h.gh.getPull = async () => {
|
||||
throw new Error("GET /pulls/1 → 502: Bad Gateway");
|
||||
};
|
||||
let thrown = null;
|
||||
await assert.rejects(h.tick({ action: "dry-run" }), (error) => {
|
||||
thrown = error;
|
||||
return /502/.test(error.message);
|
||||
});
|
||||
assert.equal(h.gh.writes.length, writes, "no GitHub write");
|
||||
assert.deepEqual(statusCalls(h), []);
|
||||
assert.equal(h.gh.contents.get(`omni-queue-state:${stateFilePath(BASE)}`)?.content, stored);
|
||||
assert.equal(failedTickRecord(thrown), null, "and nothing for main to save");
|
||||
assert.equal(newestLand(h, f.candidateSha), "success", "left for a real tick");
|
||||
});
|
||||
|
||||
test("M1: a landing that throws and whose withdrawal GitHub refuses: the failure path stores the withdrawal; the attempt count stays", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
h.world.landError = notAvailable(1);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
const state = await h.state();
|
||||
assert.equal(newestLand(h, f.candidateSha), "success", "the withdrawal was refused");
|
||||
assert.equal(state.inFlight.stage, "landing", "its own attempt: the count is the bound's record");
|
||||
assert.equal(state.inFlight.landAttempts, 1);
|
||||
assert.deepEqual(
|
||||
(state.landWithdrawals ?? []).map((w) => [w.sha, w.batchId, w.why, w.attempts]),
|
||||
[[f.candidateSha, f.batchId, "error", 1]],
|
||||
"stored by the failure path, retried by the next ticks"
|
||||
);
|
||||
});
|
||||
|
||||
// ── M4: a full withdrawal list pauses instead of dropping an entry ──
|
||||
|
||||
/** 20 stored withdrawals of App-blessed commits, then a landing refused (base-moved). */
|
||||
async function fullListThenRefusedLanding(h) {
|
||||
const f = await greenBatch(h, [1]);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
for (const sha of old) setLand(h, sha, "success");
|
||||
await storeWithdrawals(
|
||||
h,
|
||||
old.map((sha) => storedWithdrawal(sha))
|
||||
);
|
||||
h.world.baseMovesOnLand = fakeSha("manual-merge"); // the landing is refused: base-moved
|
||||
return { f, old };
|
||||
}
|
||||
|
||||
test("M4: twenty stored withdrawals and a twenty-first refusal: the base pauses, nothing is dropped, the detail names the SHA", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f, old } = await fullListThenRefusedLanding(h);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((e) => e.sha),
|
||||
old,
|
||||
"the twenty entries stay, in order; none made room"
|
||||
);
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((e) => e.attempts),
|
||||
Array(20).fill(2),
|
||||
"each retried (and refused) once more"
|
||||
);
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.ok(state.paused.detail.includes(f.candidateSha), "the commit not recorded, in full");
|
||||
assert.ok(state.paused.detail.includes(f.batchId));
|
||||
assert.match(state.paused.detail, /landWithdrawals/);
|
||||
assert.match(state.paused.detail, /resume/);
|
||||
assert.equal(state.history.at(-1).type, "error");
|
||||
assert.ok(state.history.at(-1).detail.includes(f.candidateSha));
|
||||
assert.ok(
|
||||
!state.history.some((e) => /stored withdrawal of .* dropped/.test(e.detail ?? "")),
|
||||
"no entry dropped, so no dropped event"
|
||||
);
|
||||
assert.notEqual(state.inFlight?.stage, "landing", "a paused base holds no blessed batch");
|
||||
});
|
||||
|
||||
test("M4: while paused for a full list the stored withdrawals are still retried; a resume with room continues", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f, old } = await fullListThenRefusedLanding(h);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
const pausedState = await h.state();
|
||||
assert.equal(pausedState.paused?.reason, "error");
|
||||
const recomposed = pausedState.inFlight;
|
||||
assert.notEqual(recomposed.candidateSha, f.candidateSha, "recomposed before the pause");
|
||||
|
||||
h.gh.writeError = null; // GitHub accepts the App's status writes again
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
let state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error", "still paused: the operator resumes");
|
||||
assert.equal(
|
||||
"landWithdrawals" in state,
|
||||
false,
|
||||
"every stored withdrawal was retried and written"
|
||||
);
|
||||
for (const sha of old) assert.equal(newestLand(h, sha), "failure");
|
||||
assert.deepEqual(state.inFlight, recomposed, "the batch in flight is left as is");
|
||||
|
||||
await h.tick({ action: "resume" });
|
||||
state = await h.state();
|
||||
assert.equal(state.paused, null);
|
||||
h.runCi(recomposed.candidateSha);
|
||||
await h.tick();
|
||||
state = await h.state();
|
||||
assert.ok(types(state).includes("landed"), "the queue continues");
|
||||
assert.equal(state.paused, null);
|
||||
});
|
||||
|
||||
test("M4: the failure path pauses too when a refused withdrawal finds the list full", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
for (const sha of old) setLand(h, sha, "success");
|
||||
await storeWithdrawals(
|
||||
h,
|
||||
old.map((sha) => storedWithdrawal(sha))
|
||||
);
|
||||
h.world.landError = notAvailable(1);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.ok(state.paused.detail.includes(f.candidateSha));
|
||||
assert.equal(state.landWithdrawals.length, 20);
|
||||
assert.notEqual(state.inFlight?.stage, "landing", "a paused base holds no blessed batch");
|
||||
});
|
||||
|
||||
// ── a batch that did land, met by a pause or a shadow tick ──
|
||||
|
||||
/** The landing push went through and the tick lost its save: base = candidate, PRs merged. */
|
||||
async function landedButNotSaved(h, f) {
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]);
|
||||
h.gh.branches.set(BASE, f.candidateSha);
|
||||
for (const pr of f.prs) Object.assign(h.gh.pulls.get(pr.n), { merged: true, state: "closed" });
|
||||
}
|
||||
|
||||
for (const meet of ["a manual pause", "a shadow tick"]) {
|
||||
test(`a batch that landed and lost its save, met first by ${meet}: landed recorded once, no status written on the tip`, async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1, 2]);
|
||||
await landedButNotSaved(h, f);
|
||||
if (meet === "a shadow tick") {
|
||||
h.config.mode = "shadow";
|
||||
await h.tick();
|
||||
} else {
|
||||
await h.tick({ action: "pause", reason: "incident" });
|
||||
}
|
||||
let state = await h.state();
|
||||
assert.deepEqual(statusCalls(h), [], "no status written, on the tip or anywhere");
|
||||
assert.equal(newestLand(h, f.candidateSha), "success", "the tip keeps its blessing");
|
||||
assert.equal(state.history.filter((e) => e.type === "landed").length, 1);
|
||||
assert.equal(state.history.find((e) => e.type === "landed").sha, f.candidateSha);
|
||||
assert.ok(!types(state).includes("would-land"));
|
||||
assert.equal(state.inFlight, null);
|
||||
assert.equal(state.counters.landedPrs, 2, "counters as after a normal landing");
|
||||
assert.equal(state.counters.landedBatches, 1);
|
||||
assert.deepEqual(
|
||||
state.baseChecks.map((b) => [b.baseSha, b.status]),
|
||||
[[f.candidateSha, "green"]]
|
||||
);
|
||||
if (meet === "a manual pause") assert.equal(state.paused?.reason, "manual");
|
||||
|
||||
h.clock.advance(10);
|
||||
await h.tick(meet === "a manual pause" ? { action: "resume" } : {});
|
||||
state = await h.state();
|
||||
assert.equal(state.history.filter((e) => e.type === "landed").length, 1, "once");
|
||||
assert.deepEqual(statusCalls(h), []);
|
||||
});
|
||||
}
|
||||
|
||||
test("a stored withdrawal of a candidate that did land is dropped, never written on the tip", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
await landedButNotSaved(h, f);
|
||||
// the landing threw after the push went through and its withdrawal was refused and stored
|
||||
await storeWithdrawals(h, [
|
||||
storedWithdrawal(f.candidateSha, { batchId: f.batchId, why: "error" }),
|
||||
]);
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.deepEqual(statusCalls(h), []);
|
||||
assert.equal(newestLand(h, f.candidateSha), "success");
|
||||
assert.ok(types(state).includes("landed"));
|
||||
assert.equal("landWithdrawals" in state, false, "the landed commit leaves the list");
|
||||
});
|
||||
|
||||
// ── a paused base never holds a blessed batch ──
|
||||
|
||||
test("base-red pause whose repair lane no longer runs: a base-repair batch at the landing stage goes back to testing and loses its status", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { baseRepairLabel } = h.config.lanes;
|
||||
await baseRedPause(h, [1, 2]);
|
||||
h.addPr(3);
|
||||
h.addLane(3, baseRepairLabel);
|
||||
h.world.repairs.add(3);
|
||||
await h.tick();
|
||||
const f = (await h.state()).inFlight;
|
||||
assert.equal(f.lane, "base-repair");
|
||||
h.runCi(f.candidateSha);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]); // blessed under the pause, then died
|
||||
delete h.config.lanes; // lanes turned off: the repair lane no longer runs under the pause
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused?.reason, "base-red");
|
||||
assert.equal(state.inFlight.candidateSha, f.candidateSha, "left in flight");
|
||||
assert.equal(state.inFlight.stage, "testing");
|
||||
assert.deepEqual(
|
||||
statusCalls(h).map((c) => c.args),
|
||||
[landStatus(f.batchId, f.candidateSha, { why: LEFT_LANDING })]
|
||||
);
|
||||
assert.deepEqual(h.world.lands, []);
|
||||
});
|
||||
|
||||
test("M1 through main(), shadow mode (the shipped config): a tick that throws after it left a stored landing withdraws the status and saves the batch at testing with its error", async (t) => {
|
||||
const q = await mainAgainstFake(t);
|
||||
const now = new Date().toISOString();
|
||||
const candidateSha = fakeSha("main-candidate");
|
||||
q.fake.refs.set(`heads/${q.base}`, fakeSha("main-tip"));
|
||||
const seeded = initialState(q.base, now);
|
||||
seeded.inFlight = {
|
||||
batchId: "b1",
|
||||
kind: "batch",
|
||||
candidateRef: candidateRef(q.base, "b1", candidateSha),
|
||||
candidateSha,
|
||||
baseSha: fakeSha("main-base"),
|
||||
treeSha: null,
|
||||
prs: [{ n: 7, head: fakeSha("main-head-7"), title: "PR 7", author: "dev7", labeledAt: now }],
|
||||
stage: "landing", // a live tick blessed it and died before its save
|
||||
landAttempts: 1,
|
||||
startedAt: now, // its checks have not reported: this tick finds them pending
|
||||
attempts: 0,
|
||||
bisect: null,
|
||||
};
|
||||
q.fake.commitFiles({
|
||||
branch: STATE_BRANCH,
|
||||
files: { [stateFilePath(q.base)]: `${JSON.stringify(seeded, null, 2)}\n` },
|
||||
message: "seed: a batch at the landing stage",
|
||||
});
|
||||
q.fake.statuses.set(candidateSha, [
|
||||
{
|
||||
context: "omni-queue/land",
|
||||
state: "success",
|
||||
description: "omni-queue b1: checks green",
|
||||
creator: { login: "omni-queue[bot]", type: "Bot" },
|
||||
created_at: now,
|
||||
},
|
||||
]);
|
||||
q.failNextSnapshot(); // the tick throws after it handled the batch in flight
|
||||
assert.equal(await q.tick(), 1, "the tick still fails");
|
||||
const landStatusesOf = (sha) =>
|
||||
(q.fake.statuses.get(sha) ?? [])
|
||||
.filter((s) => s.context === "omni-queue/land")
|
||||
.map((s) => [s.state, s.creator?.login]);
|
||||
assert.deepEqual(landStatusesOf(candidateSha), [
|
||||
["failure", "omni-queue[bot]"],
|
||||
["success", "omni-queue[bot]"],
|
||||
]);
|
||||
assert.equal(
|
||||
q.fake.statuses.get(candidateSha)[0].description,
|
||||
`omni-queue b1: landing withdrawn (${LEFT_LANDING})`
|
||||
);
|
||||
const state = q.state();
|
||||
assert.equal(state.inFlight.stage, "testing", "saved with the error event");
|
||||
assert.equal("landAttempts" in state.inFlight, false);
|
||||
assert.equal(state.history.at(-1).type, "error");
|
||||
assert.match(state.history.at(-1).detail, /listing broke/);
|
||||
assert.equal(state.failedTicks, 1);
|
||||
assert.ok(!q.errors.some((line) => /could not record the error/.test(line)), q.errors.join("\n"));
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user