mirror of
https://github.com/diegosouzapw/OmniRoute.git
synced 2026-11-18 07:58:27 +00:00
fix(ci): omni-queue keeps every refused withdrawal and leaves an operator's pause alone
Task 13b, fix round 1, rulings V2 and V3. V2: a refused withdrawal that found landWithdrawals full was kept only in the pause detail and one error event (review probe 3: after the drain and the resume the commit kept the App's success for ever). It now goes to the new optional state field landWithdrawalsOverflow (same entries, at most LAND_WITHDRAWALS_LIMIT, one per SHA across both lists), retried by the same code on every tick in every mode (retryList) and dropped like the main list (dropWithdrawals, the failure path's retired SHAs). storeWithdrawalIn says which list holds a refusal; recordFullWithdrawals replaces pauseForUnrecordedWithdrawals: the full-list pause stays for a base that is not paused, its detail (starting WITHDRAWALS_FULL_DETAIL, "landWithdrawals is full") gives the size of both lists, and only a refusal that fits in neither is named in full, in the detail and as the error event's sha. Contract: validateState checks landWithdrawalsOverflow when present (withdrawalListErrors: the same entry rules, at most 20, no SHA of the main list). A state without it loads; the validators of05b9bd4and of origin/release/v3.8.54 return no error for a state that has it (checked by importing them: they ignore unknown top-level fields, and the old tick carries the field along without retrying it); the runbook's rollback list says to empty it before such a rollback. V3: a base already paused keeps its pause (reason, detail, since) when the list fills: the condition is only an error event, the refusal goes to the overflow; a full-list pause of its own gets the commits kept nowhere appended. The "(was paused: ...)" quoting is gone. Runbook: Who lands, State (the new field), Events (error row), shadow mode's withdrawals, Before a mode change (both lists empty; the error event of a commit kept nowhere), Kill switch, cut-over step 9, rollback (first line and the new clean-up item), Incidents ("Paused: landWithdrawals is full": what a resume does and what to check first; the hand edit only for the last resort, with the entry format and the order of steps). Re-aimed tests (tick.test.mjs), each keeping its earlier checks: - "M4: twenty stored withdrawals and a twenty-first refusal: the base pauses, nothing is dropped, the detail names the SHA" is now "... the refusal is kept in the overflow": the refusal is asserted in landWithdrawalsOverflow and as the event's sha; the detail gives the counts and no longer names a stored commit. - "M4: while paused for a full list the stored withdrawals are still retried; a resume with room continues" is now "... both lists are still retried ...": also asserts the overflow drained and the twenty-first commit at failure. - "M4: the failure path pauses too when a refused withdrawal finds the list full" now also asserts the refusal in the overflow. New tests, failed on6b20bd7unless marked: - "M4 (review probe 3): after the drain and the resume, the twenty-first commit is withdrawn too and both lists are empty" - "M4: both lists full: the commit that fits nowhere is named in the pause detail and in the event's sha" - "M4 (review probe 6): a base already paused by hand keeps its pause when the list is full; one resume lifts that pause" - state.test.mjs: "load: landWithdrawalsOverflow is absent in older documents, and accepted up to LAND_WITHDRAWALS_LIMIT entries, one per SHA across both lists" (passed at once: an unknown field was already carried), "load: a malformed landWithdrawalsOverflow is refused, and never saved", "a top-level field the validator does not know is ignored: how code before landWithdrawalsOverflow loads a state that has it" (passed at once: pins the rule older code relies on).
This commit is contained in:
@@ -231,11 +231,15 @@ times; after the third, `main()` records the error without anything the tick did
|
||||
record): the next tick reads the stored state again.
|
||||
|
||||
The list holds at most 20 entries (`LAND_WITHDRAWALS_LIMIT` in
|
||||
`scripts/merge-queue/contracts.mjs`). Nothing is ever dropped from it: when it is full and GitHub
|
||||
refuses one more withdrawal of a commit it does not hold, the base is paused with reason `error`
|
||||
and the commit is named in full in the pause detail, which starts with
|
||||
`land-status withdrawals not recorded` (see [Incidents](#incidents)). The stored entries stay and
|
||||
are still retried on every tick of the paused base.
|
||||
`scripts/merge-queue/contracts.mjs`). Nothing is ever dropped from it. A refusal that finds it full
|
||||
is stored in the state field `landWithdrawalsOverflow` (same entries, at most 20 more), and both
|
||||
lists are retried by the same code on every tick, in every mode. A full main list means GitHub is
|
||||
refusing the App's status writes, so a base that is not paused yet is paused with reason `error`; the
|
||||
detail starts with `landWithdrawals is full` and says how many entries each list holds. A base
|
||||
already paused keeps its own pause (reason, detail, time); an `error` event records the full list.
|
||||
Only a refusal that fits in neither list is kept nowhere: its commit is named in full in the pause
|
||||
detail (a pause of this kind gets it appended) and in that `error` event (see
|
||||
[Incidents](#incidents)).
|
||||
|
||||
In live mode:
|
||||
|
||||
@@ -508,7 +512,7 @@ tick's `checks-red`; when no re-run was accepted, the candidate is classified in
|
||||
The queue state lives on the branch `omni-queue-state` (`STATE_BRANCH`), one JSON file per base:
|
||||
`state/<sanitized-base>.json`, for example `state/release-v3.8.55.json`. It holds the pause, the
|
||||
candidate in flight (refs, SHAs, PRs, stage, attempts, bisection), the last base checks, the
|
||||
shadow bookkeeping, counters and the last 200 history events. Two fields belong to the land status
|
||||
shadow bookkeeping, counters and the last 200 history events. Three fields belong to the land status
|
||||
(see [Who lands](#who-lands-landingactor)):
|
||||
|
||||
- `inFlight.landAttempts`: the number of ticks in a row that blessed the batch without reaching a
|
||||
@@ -518,8 +522,9 @@ shadow bookkeeping, counters and the last 200 history events. Two fields belong
|
||||
pauses instead of blessing it again.
|
||||
- `landWithdrawals`: present only when it is not empty. Each entry is a commit whose `failure` land
|
||||
status GitHub refused to write (`sha`, `batchId`, `candidateSha`, `why`, `since`, `attempts`),
|
||||
oldest first, one per SHA, at most 20. Later ticks retry them. A refusal that a full list cannot
|
||||
keep pauses the base instead of replacing an entry.
|
||||
oldest first, one per SHA, at most 20. Later ticks retry them.
|
||||
- `landWithdrawalsOverflow`: the same entries, for the refusals that found `landWithdrawals` full;
|
||||
at most 20, no SHA of the main list, retried alike, present only when it is not empty.
|
||||
|
||||
A document without these fields loads as before, and code that predates them loads a document that
|
||||
has them (`validateState` ignores fields it does not know), but that code does not act on them (see
|
||||
@@ -589,7 +594,7 @@ when this table and `EVENT_TYPES` differ.
|
||||
| `paused` | A manual pause (`npm run queue -- pause`, or `action=pause`). | `npm run queue -- resume <base>` when ready. |
|
||||
| `resumed` | A manual resume (`manual resume (was <reason>)`), a `base-red` pause lifted because the base tip moved (`base moved <sha12> → <sha12> after a base-red pause`), or, in live mode, a `base-red` pause lifted because a PR of the base-repair lane landed (`base repaired by #<n>`). | Nothing. |
|
||||
| `tree-divergence` | `squash-api` only: the base tree after the merges differs from the candidate tree, or an open divergence issue stops a landing. The base is paused with reason `tree-divergence`. | Compare the trees, fix the base, close the issue, resume (see [Incidents](#incidents)). |
|
||||
| `error` | A tick failed; a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused; a refused withdrawal the full `landWithdrawals` list (20) cannot keep, which pauses the base (`land-status withdrawals not recorded: …`); a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed. The detail says which. | Read the detail: each pause detail has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
|
||||
| `error` | A tick failed; a landing pause (reason `error`: land token, landing refused, composition mismatch, queue refs, base not found, attempts limit); a land-status withdrawal GitHub refused; a refused withdrawal that found the `landWithdrawals` list (20) full, kept in `landWithdrawalsOverflow` (`landWithdrawals is full …`; it pauses a base that is not paused); a candidate gone from origin (`… is gone from origin; recomposing …`), whose PRs are recomposed. The detail says which. | Read the detail: each pause detail has an entry in [Incidents](#incidents). A landing that fails on every tick ends in the pause after three attempts. |
|
||||
| `queue-stalled` | The queue stopped making progress (`assessHealth` in `scripts/merge-queue/dashboard.mjs`): paused for more than 120 minutes, a candidate in flight for more than `checksTimeoutMinutes` + 60 (a batch held by a release freeze is waiting, not stalled; once the hold ends, counted from its end: `inFlight.freezeHold`), eligible PRs and nothing started for more than `batch.maxWaitMinutes` + 30, or three ticks in a row that failed (`failedTicks`, counted by the failure path and deleted by the next tick that completes; `error` events appended by ticks that completed do not count). Recorded once per episode, with `stalled` in the state and one comment on the dashboard issue that mentions the approvers. Until that comment is posted (`notified` in `stalled`), every tick, completed or failed, tries again; a failed tick creates the dashboard issue when there is none. The detail lists the reasons. | Read the reasons; each points at its cause (the pause, the candidate's checks, the eligible PRs, the failing tick's `error` events). |
|
||||
|
||||
## Configuration keys
|
||||
@@ -704,7 +709,8 @@ Shadow mode observes. The PRs it reads are the ones Mergify still lands, so it l
|
||||
Shadow mode writes a `failure` land status only to withdraw one: on the commits of a batch stored
|
||||
at the landing stage whose land status is `success` by the rule of
|
||||
[Who lands](#who-lands-landingactor) or cannot be read (unknown: withdrawn anyway), and on the
|
||||
commits of stored withdrawals (`landWithdrawals`). Only live mode stores a batch at the landing
|
||||
commits of stored withdrawals (`landWithdrawals`, `landWithdrawalsOverflow`). Only live mode
|
||||
stores a batch at the landing
|
||||
stage. A shadow tick stores a withdrawal too, when GitHub refuses its own withdrawal of such a
|
||||
batch, but every stored withdrawal comes from a batch live mode blessed: a queue that was never
|
||||
live writes none.
|
||||
@@ -748,8 +754,8 @@ change takes effect when it is merged there.
|
||||
|
||||
A live tick can leave the land status `success` on commits of a batch that did not land: a tick
|
||||
that died after the blessing leaves its batch stored at the in-flight stage `landing`, and a
|
||||
withdrawal GitHub refused is stored in `landWithdrawals` (see
|
||||
[Who lands](#who-lands-landingactor)). Every later tick withdraws both, in shadow mode too (a
|
||||
withdrawal GitHub refused is stored in `landWithdrawals` or `landWithdrawalsOverflow` (see
|
||||
[Who lands](#who-lands-landingactor)). Every later tick withdraws them, in shadow mode too (a
|
||||
shadow tick sends a batch stored at the landing stage back to `testing` and turns its `success`
|
||||
into `failure` first), but no tick runs while the kill switch is off. So before switching a live
|
||||
queue to shadow (or shadow to live), turning the kill switch off or rolling back:
|
||||
@@ -771,14 +777,13 @@ queue to shadow (or shadow to live), turning the kill switch off or rolling back
|
||||
batch at stage `landing` with the base first if the history matters.
|
||||
|
||||
2. Check that no batch is at stage `landing` (`npm run queue -- status --base <base>` prints
|
||||
`stage <stage>` for the batch in flight) and that `landWithdrawals` is empty: the field is absent
|
||||
from `state/<sanitized-base>.json` on `omni-queue-state` (an empty list leaves no field). While
|
||||
entries remain, wait for the next ticks to retry them and check again. Nothing is dropped from
|
||||
that list: a refused withdrawal that a full list (20 entries) cannot keep pauses the base with
|
||||
reason `error` and a detail that starts with `land-status withdrawals not recorded` and names
|
||||
each such commit in full. If the pause detail says so, handle it first (see
|
||||
[Incidents](#incidents), "Paused: land-status withdrawals not recorded"): each commit it names
|
||||
may still carry the App's `success`.
|
||||
`stage <stage>` for the batch in flight) and that both `landWithdrawals` and
|
||||
`landWithdrawalsOverflow` are empty: the fields are absent from `state/<sanitized-base>.json` on
|
||||
`omni-queue-state` (an empty list leaves no field). While entries remain, wait for the next ticks
|
||||
to retry them and check again. Also read the `error` events of the history
|
||||
(`npm run queue -- history --base <base>`) for one that starts with `landWithdrawals is full` and
|
||||
says `kept nowhere`: each commit it names may still carry the App's `success` (see
|
||||
[Incidents](#incidents), "Paused: landWithdrawals is full").
|
||||
|
||||
3. Only then make the change. After a mode change, resume each base
|
||||
(`npm run queue -- resume <base>`).
|
||||
@@ -793,8 +798,8 @@ other run. It is not an immediate stop: the variable
|
||||
is read only when a job starts, so a tick already running (and possibly one already pending)
|
||||
finishes and can still land a green candidate. Pause the base first
|
||||
(`npm run queue -- pause <base>`) and check that no batch is at stage `landing` and that
|
||||
`landWithdrawals` is empty (see [Before a mode change or a stop](#before-a-mode-change-or-a-stop)):
|
||||
with the variable off no tick runs, so nothing withdraws a land status left by a tick that died.
|
||||
`landWithdrawals` and `landWithdrawalsOverflow` are empty (see
|
||||
[Before a mode change or a stop](#before-a-mode-change-or-a-stop)): with the variable off no tick runs, so nothing withdraws a land status left by a tick that died.
|
||||
Then flip the variable, cancel the in-progress and pending `omni-queue` runs in the Actions tab,
|
||||
and confirm with `npm run queue -- history` and the base tip that nothing landed after the cut.
|
||||
Forks never inherit repository variables.
|
||||
@@ -1512,30 +1517,45 @@ blessing. The status of the batch's commits was withdrawn. Read the last `error`
|
||||
that outlasts three ticks ends here too: a 403 that says `rate limit` is retried, not treated as a
|
||||
refused token. Once the cause is gone, `npm run queue -- resume <base>` recomposes the batch.
|
||||
|
||||
**Paused: land-status withdrawals not recorded.** The pause detail
|
||||
`land-status withdrawals not recorded: landWithdrawals already holds 20 entries, so the refused withdrawal of <landing.statusContext> on <sha> (<batchId>) is stored nowhere (nothing was dropped). …`
|
||||
means GitHub refused to write the `failure` land status on that commit while the list of stored
|
||||
withdrawals was full, so no tick will retry it. Its newest land status by the App may still be
|
||||
`success`, which the release ruleset accepts: the commit can be pushed or merged into the base by
|
||||
hand. A pause of this kind already in place gets the next such commits appended
|
||||
(`; also not recorded: …`); a pause it replaced is quoted at the end (`(was paused: …)`). Check,
|
||||
in order:
|
||||
**Paused: landWithdrawals is full.** A pause detail that starts with
|
||||
`landWithdrawals is full (<n> of 20; landWithdrawalsOverflow holds <m> of 20)` means GitHub refused
|
||||
to write the App's `failure` land status while 20 refused withdrawals were already stored: GitHub
|
||||
is refusing the App's status writes, so the queue stopped blessing. Every refused withdrawal is
|
||||
kept, in `landWithdrawals` or in `landWithdrawalsOverflow`, and both lists are retried on every
|
||||
tick of the paused base. A base that was already paused (a manual pause before a mode change, for
|
||||
example) keeps its own pause: the same condition is then only an `error` event that starts with
|
||||
`landWithdrawals is full`. Check, in order:
|
||||
|
||||
1. Why GitHub refuses the App's status writes: the run's log names the error of each refused
|
||||
write, and so does an `error` event (`<batchId>: could not withdraw …`, or the detail of the
|
||||
failed tick's own `error` event when the tick threw). A revoked
|
||||
`statuses: write` permission of the App, an uninstalled App or a GitHub incident are the usual
|
||||
causes.
|
||||
2. Once that is fixed, the next ticks of the paused base retry the stored withdrawals; wait until
|
||||
`landWithdrawals` is gone from `state/<sanitized-base>.json` (or has room).
|
||||
3. The land status of each commit the detail names
|
||||
(`gh api repos/<owner>/<repo>/commits/<sha>/statuses`, newest first, context
|
||||
`<landing.statusContext>`, creator `<appSlug>[bot]`). Where the App's newest is `success`, add
|
||||
an entry for that commit to `landWithdrawals` with an ordinary commit on `omni-queue-state`
|
||||
(same fields as the others; with the queue-refs ruleset applied, disable it first and enable it
|
||||
again after): the next tick writes `failure` there.
|
||||
failed tick's own `error` event when the tick threw). A revoked `statuses: write` permission of
|
||||
the App, an uninstalled App or a GitHub incident are the usual causes.
|
||||
2. Once that is fixed, wait until both lists are gone from `state/<sanitized-base>.json` on
|
||||
`omni-queue-state` (the next ticks write the `failure` statuses). Nothing else needs a person.
|
||||
3. Then `npm run queue -- resume <base>`. The resume lifts whatever pause is there: read
|
||||
`paused.reason` and `paused.detail` first, and resume only when this is the pause to lift (an
|
||||
operator's own pause, kept as it was, needs its own reason to be over). The batch in flight, if
|
||||
any, is then followed again from where it was: its checks are read and, green, it is blessed
|
||||
and landed as usual.
|
||||
|
||||
Then `npm run queue -- resume <base>`: the batch in flight, if any, goes on from where it was.
|
||||
The last resort: when both lists were full too, a refused withdrawal is kept nowhere. Its commit is
|
||||
then named in full, as `<sha> (<batchId>)` after the words `Kept nowhere (both lists full)`, in
|
||||
`paused.detail` of `state/<sanitized-base>.json` and in the `sha` and detail of that `error`
|
||||
event. Read it in the state file itself: the dashboard and `npm run queue -- status` cut
|
||||
long details. That commit's App status may stay `success` and nothing retries it; a person has to
|
||||
put it back in a list, once a list has room (after step 2):
|
||||
|
||||
1. The base is already paused (this pause); keep it paused.
|
||||
2. Disable the `omni-queue refs protection` ruleset (Settings → Rules → Rulesets), so an ordinary
|
||||
commit can change `omni-queue-state`.
|
||||
3. In `state/<sanitized-base>.json`, append to `landWithdrawals` (or to `landWithdrawalsOverflow`;
|
||||
at most 20 entries in each, and a SHA in one list only) one entry per named commit:
|
||||
`{"sha": "<40-hex sha>", "batchId": "<batchId>", "candidateSha": "<40-hex sha>", "why": "left the landing stage", "since": "<ISO 8601 time, e.g. 2026-11-16T12:00:00.000Z>", "attempts": 1}`,
|
||||
where `candidateSha` is the commit's own SHA when you do not know the batch's candidate. Commit
|
||||
that file alone; an invalid document makes every tick fail (see [State](#state)).
|
||||
4. Enable the ruleset again.
|
||||
5. `npm run queue -- resume <base>` when the next tick has written the `failure` (the entry is
|
||||
gone), after the checks of step 3 above.
|
||||
|
||||
**Labels with no effect.** If labeled PRs stay `approval-missing` and no `omni-queue/approval`
|
||||
status appears on their heads, the `approval` job is not running: look for failed
|
||||
@@ -1591,8 +1611,8 @@ shadow mode on the real repository. In order:
|
||||
`live mode would pause: queue refs unprotected (…)` and no
|
||||
`<composition check|queue refs check> could not run (…): live mode would fail the tick`; and
|
||||
no shadow batch was paused for a composition mismatch;
|
||||
- for this and any later mode change: each base paused, no batch at stage `landing` and an
|
||||
empty `landWithdrawals` (see
|
||||
- for this and any later mode change: each base paused, no batch at stage `landing` and empty
|
||||
`landWithdrawals` and `landWithdrawalsOverflow` (see
|
||||
[Before a mode change or a stop](#before-a-mode-change-or-a-stop)).
|
||||
|
||||
The queue cannot land that pull request (it is still in shadow mode): bless its head, then merge
|
||||
@@ -1608,7 +1628,7 @@ shadow mode on the real repository. In order:
|
||||
then merge it, as in step 8.
|
||||
|
||||
Rollback: first pause each base and check that no batch is at stage `landing` and that
|
||||
`landWithdrawals` is empty (see [Before a mode change or a stop](#before-a-mode-change-or-a-stop)).
|
||||
`landWithdrawals` and `landWithdrawalsOverflow` are empty (see [Before a mode change or a stop](#before-a-mode-change-or-a-stop)).
|
||||
Then set OMNI_QUEUE_ENABLED to `false`. That stops new ticks only: cancel the in-progress and
|
||||
pending `omni-queue` runs in the Actions tab (or wait for them to end), and check with
|
||||
`npm run queue -- history` and each base tip that nothing landed after the cut before declaring
|
||||
@@ -1659,5 +1679,12 @@ the rollback done (see [Kill switch](#kill-switch)). Then:
|
||||
the first new tick, check the newest `omni-queue/land` status of each blessed SHA and bless it
|
||||
again where it reads `failure`. A batch stored at the landing stage also keeps its
|
||||
`inFlight.landAttempts` across both moves, so after the roll-forward it can reach the attempts
|
||||
limit with fewer than three new attempts. A `landAttempts` that a failed tick left at the stage
|
||||
`testing` is ignored by that code and replaced at the batch's next landing; nothing to clean.
|
||||
limit with fewer than three new attempts;
|
||||
- rolling the code back to a version from before `landWithdrawalsOverflow` (and before
|
||||
`inFlight.landAttempts` was kept at the stage `testing` by a failed tick) needs that field to be
|
||||
empty first: that code loads the document and carries the field along (`validateState` ignores
|
||||
fields it does not know) but never retries its entries, so a blessing stored there would stay
|
||||
until the roll-forward. Wait until the field is gone (the paused base's ticks retry it), or move
|
||||
its entries into `landWithdrawals` when that list has room, with an ordinary commit as above. A
|
||||
`landAttempts` left at the stage `testing` is ignored by that code and replaced at the batch's
|
||||
next landing; nothing to clean.
|
||||
|
||||
@@ -33,8 +33,10 @@ export const HISTORY_LIMIT = 200;
|
||||
export const BASE_CHECK_LIMIT = 10;
|
||||
/**
|
||||
* Land-status withdrawals GitHub refused, kept for the next ticks to retry (state.landWithdrawals,
|
||||
* one per SHA). When full, nothing is dropped: a refusal the list cannot keep pauses the base with
|
||||
* reason `error`, the commit named in the pause detail (tick pauseForUnrecordedWithdrawals).
|
||||
* one per SHA), and the size of their overflow (state.landWithdrawalsOverflow, the refusals that
|
||||
* found the main list full). Nothing is dropped: a full main list pauses a base that is not paused
|
||||
* yet with reason `error`; a refusal that fits in neither list is named in the pause detail and in
|
||||
* an `error` event (tick recordFullWithdrawals).
|
||||
*/
|
||||
export const LAND_WITHDRAWALS_LIMIT = 20;
|
||||
|
||||
@@ -309,6 +311,11 @@ const SAFE_SEGMENT_RE = /^[A-Za-z0-9._-]+$/;
|
||||
* @property {LandWithdrawal[]} [landWithdrawals] oldest first, at most LAND_WITHDRAWALS_LIMIT,
|
||||
* one per SHA; absent when there is none (and in documents
|
||||
* written before it existed)
|
||||
* @property {LandWithdrawal[]} [landWithdrawalsOverflow] the refusals that found landWithdrawals
|
||||
* full, retried alike; oldest first, at most
|
||||
* LAND_WITHDRAWALS_LIMIT, no SHA of landWithdrawals; absent
|
||||
* when there is none (and in documents written before it
|
||||
* existed; code before it carries it along and ignores it)
|
||||
* @property {number} [batchWindow] config.batch.adaptive only: the size of the next root
|
||||
* batch (eligibility.currentBatchWindow reads it, clamped to
|
||||
* [minSize, maxSize]); absent until the window first moves
|
||||
@@ -531,7 +538,12 @@ export function validateState(state) {
|
||||
if (!isRecord(c) || counterKeys.some((k) => !isCount(c[k]))) errors.push("counters");
|
||||
if (!Array.isArray(state.history)) errors.push("history");
|
||||
else if (state.history.some((e) => !isValidEvent(e))) errors.push("history[]");
|
||||
if (state.landWithdrawals !== undefined) errors.push(...landWithdrawalsErrors(state));
|
||||
if (state.landWithdrawals !== undefined) {
|
||||
errors.push(...withdrawalListErrors(state, "landWithdrawals"));
|
||||
}
|
||||
if (state.landWithdrawalsOverflow !== undefined) {
|
||||
errors.push(...withdrawalListErrors(state, "landWithdrawalsOverflow"));
|
||||
}
|
||||
if (
|
||||
state.batchWindow !== undefined &&
|
||||
!(Number.isSafeInteger(state.batchWindow) && state.batchWindow >= 1)
|
||||
@@ -578,13 +590,19 @@ function isStalledEpisode(episode) {
|
||||
);
|
||||
}
|
||||
|
||||
/** Optional state.landWithdrawals: a bounded list of LandWithdrawal, one per SHA. */
|
||||
function landWithdrawalsErrors({ landWithdrawals: list }) {
|
||||
if (!Array.isArray(list) || list.length > LAND_WITHDRAWALS_LIMIT) return ["landWithdrawals"];
|
||||
/**
|
||||
* Optional state.landWithdrawals and state.landWithdrawalsOverflow: each a bounded list of
|
||||
* LandWithdrawal, one entry per SHA across both lists (an overflow entry may not repeat a SHA of
|
||||
* the main list).
|
||||
*/
|
||||
function withdrawalListErrors(state, field) {
|
||||
const list = state[field];
|
||||
if (!Array.isArray(list) || list.length > LAND_WITHDRAWALS_LIMIT) return [field];
|
||||
const errors = [];
|
||||
const seen = new Set();
|
||||
const main = field === "landWithdrawals" || !Array.isArray(state.landWithdrawals);
|
||||
const seen = new Set(main ? [] : state.landWithdrawals.map((w) => w?.sha));
|
||||
list.forEach((w, i) => {
|
||||
const where = `landWithdrawals[${i}]`;
|
||||
const where = `${field}[${i}]`;
|
||||
if (!isRecord(w)) return errors.push(where);
|
||||
if (!SHA_RE.test(w.sha ?? "") || seen.has(w.sha)) errors.push(`${where}.sha`);
|
||||
seen.add(w.sha);
|
||||
|
||||
@@ -88,8 +88,10 @@
|
||||
// rethrowing (settleFailure), unless it landed it or made its own landing attempt (that
|
||||
// attempt withdrew its own blessing and keeps its count). A `failure` GitHub refuses to write
|
||||
// is kept in state.landWithdrawals and retried by the next ticks (retryWithdrawals); when that
|
||||
// list is full (LAND_WITHDRAWALS_LIMIT) nothing is dropped: the base pauses with reason
|
||||
// "error" and the pause detail names the commit (pauseForUnrecordedWithdrawals).
|
||||
// list is full (LAND_WITHDRAWALS_LIMIT) nothing is dropped: the refusal goes to
|
||||
// state.landWithdrawalsOverflow, retried alike, and a base that is not paused pauses with
|
||||
// reason "error" (recordFullWithdrawals); only a refusal that fits in neither list is named,
|
||||
// in full, in the pause detail and in an `error` event.
|
||||
// Before landing (or composing a bisection segment) every PR is re-read and re-run through the
|
||||
// eligibility rules: block labels, approval status on the head, protected approval and an open
|
||||
// release-freeze (which holds the green batch in flight instead of landing it). Then the candidate
|
||||
@@ -538,8 +540,8 @@ class Tick {
|
||||
*/
|
||||
this.refused = [];
|
||||
this.retired = [];
|
||||
/** Refused withdrawals a full landWithdrawals list could not keep (pauseIfUnrecorded). */
|
||||
this.unrecorded = [];
|
||||
/** Refusals that found landWithdrawals full, {…, list} (recordIfFull). */
|
||||
this.fullRefusals = [];
|
||||
/** {ref, result} of this tick's queue-refs protection read (queueRefsProtection). */
|
||||
this.queueRefs = null;
|
||||
}
|
||||
@@ -780,7 +782,7 @@ class Tick {
|
||||
} else if (this.s.paused) this.note(`paused (${this.s.paused.reason}): nothing composed`);
|
||||
|
||||
await this.withdrawAbandoned();
|
||||
await this.pauseIfUnrecorded();
|
||||
await this.recordIfFull();
|
||||
await this.cleanupRefs(loadedRefs);
|
||||
this.endFreezeHold();
|
||||
const undelivered = this.trackHealth(queue.entries);
|
||||
@@ -1760,21 +1762,23 @@ class Tick {
|
||||
}
|
||||
|
||||
/**
|
||||
* End of a tick: the withdrawals GitHub refused this tick that a full landWithdrawals list could
|
||||
* not keep (storeWithdrawal) pause the base with reason `error`, naming each commit (pure
|
||||
* pauseForUnrecordedWithdrawals). The pause leaves a landing stage (a paused base holds no
|
||||
* blessed batch), so withdrawAbandoned runs once more; a refusal there is named as well.
|
||||
* End of a tick: the withdrawals GitHub refused this tick while landWithdrawals was full
|
||||
* (storeWithdrawal: kept in the overflow, or nowhere) are recorded by the pure
|
||||
* recordFullWithdrawals: a base that is not paused pauses with reason `error`, a paused one keeps
|
||||
* its pause, and an `error` event says which commits. A new pause leaves a landing stage (a
|
||||
* paused base holds no blessed batch), so withdrawAbandoned runs once more; a refusal there is
|
||||
* recorded the same way.
|
||||
*/
|
||||
async pauseIfUnrecorded() {
|
||||
for (let pass = 0; pass < 2 && this.unrecorded.length; pass++) {
|
||||
const unrecorded = this.unrecorded;
|
||||
this.unrecorded = [];
|
||||
this.s = pauseForUnrecordedWithdrawals(this.s, unrecorded, {
|
||||
async recordIfFull() {
|
||||
for (let pass = 0; pass < 2 && this.fullRefusals.length; pass++) {
|
||||
const refusals = this.fullRefusals;
|
||||
this.fullRefusals = [];
|
||||
this.s = recordFullWithdrawals(this.s, refusals, {
|
||||
at: this.now(),
|
||||
url: this.deps.runUrl ?? undefined,
|
||||
statusContext: this.config.landing.statusContext,
|
||||
});
|
||||
this.note(`paused: ${this.s.paused.detail}`);
|
||||
this.note(this.s.history.at(-1).detail);
|
||||
await this.withdrawAbandoned();
|
||||
}
|
||||
}
|
||||
@@ -1809,41 +1813,49 @@ class Tick {
|
||||
|
||||
/**
|
||||
* Keep a `failure` GitHub refused to write on `sha` for the next ticks (retryWithdrawals), by
|
||||
* the pure storeWithdrawalIn: one entry per SHA, at most LAND_WITHDRAWALS_LIMIT. Nothing is ever
|
||||
* dropped: a refusal the full list cannot keep is held in this.unrecorded, and the end of the
|
||||
* tick pauses the base naming it (pauseIfUnrecorded). Every refusal is also kept in
|
||||
* this.refused for main()'s failure path.
|
||||
* the pure storeWithdrawalIn: one entry per SHA, in state.landWithdrawals while it has room,
|
||||
* else in state.landWithdrawalsOverflow. Nothing is ever dropped. A refusal that did not go to
|
||||
* the main list (overflow, or nowhere when both are full) is held in this.fullRefusals for the
|
||||
* end of the tick (recordIfFull). Every refusal is also kept in this.refused for main()'s
|
||||
* failure path.
|
||||
*/
|
||||
storeWithdrawal(f, sha, why) {
|
||||
const refusal = { sha, batchId: f.batchId, candidateSha: f.candidateSha, why, at: this.now() };
|
||||
this.refused.push(refusal);
|
||||
const { state, stored } = storeWithdrawalIn(this.s, refusal);
|
||||
this.setLandWithdrawals(state.landWithdrawals ?? []);
|
||||
if (stored) return;
|
||||
this.unrecorded.push(refusal);
|
||||
const { state, list } = storeWithdrawalIn(this.s, refusal);
|
||||
for (const field of WITHDRAWAL_LISTS) this.setWithdrawalList(field, state[field] ?? []);
|
||||
if (list === "landWithdrawals") return;
|
||||
this.fullRefusals.push({ ...refusal, list });
|
||||
this.note(
|
||||
`${f.batchId}: withdrawal of ${head12(sha)} not stored: landWithdrawals holds ` +
|
||||
`${LAND_WITHDRAWALS_LIMIT} entries; the base pauses at the end of the tick`
|
||||
`${f.batchId}: withdrawal of ${head12(sha)} ` +
|
||||
(list ? "kept in landWithdrawalsOverflow" : "kept nowhere: both lists are full") +
|
||||
"; the full list is recorded at the end of the tick"
|
||||
);
|
||||
}
|
||||
|
||||
/** state.landWithdrawals = `list`; an empty list leaves no field (documents stay as before). */
|
||||
setLandWithdrawals(list) {
|
||||
if (list.length) this.s.landWithdrawals = list;
|
||||
else delete this.s.landWithdrawals;
|
||||
/** this.s[field] = `list`; an empty list leaves no field (documents stay as before). */
|
||||
setWithdrawalList(field, list) {
|
||||
if (list.length) this.s[field] = list;
|
||||
else delete this.s[field];
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored withdrawals of `shas` go: those commits are blessed on purpose (markLanding, a
|
||||
* manual bless) or landed (afterLanded), so withdrawing them would be wrong.
|
||||
* The stored withdrawals of `shas` go, from both lists: those commits are blessed on purpose
|
||||
* (markLanding, a manual bless) or landed (afterLanded), so withdrawing them would be wrong.
|
||||
*/
|
||||
dropWithdrawals(shas) {
|
||||
const drop = new Set(shas);
|
||||
this.setLandWithdrawals((this.s.landWithdrawals ?? []).filter((w) => !drop.has(w.sha)));
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
this.setWithdrawalList(
|
||||
field,
|
||||
(this.s[field] ?? []).filter((w) => !drop.has(w.sha))
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Retry each stored withdrawal (storeWithdrawal): read the land status that decides a
|
||||
* Retry each stored withdrawal of both lists (storeWithdrawal; the main list, then the overflow,
|
||||
* by the same code): read the land status that decides a
|
||||
* withdrawal on its commit (newestAppLandStatus); `success` → write `failure` again. The entry
|
||||
* goes once that write succeeds, or when that status is not `success` (withdrawn since, or none:
|
||||
* nothing of the queue's to withdraw; a status by a readable other creator changes nothing, and
|
||||
@@ -1856,8 +1868,15 @@ class Tick {
|
||||
* so it reads nothing); never in a dry-run (which writes nothing).
|
||||
*/
|
||||
async retryWithdrawals() {
|
||||
const stored = this.s.landWithdrawals ?? [];
|
||||
if (!stored.length || this.dryRun) return;
|
||||
if (this.dryRun) return;
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
const stored = this.s[field] ?? [];
|
||||
if (stored.length) this.setWithdrawalList(field, await this.retryList(stored));
|
||||
}
|
||||
}
|
||||
|
||||
/** retryWithdrawals for one list: returns the entries kept (each with its attempts counted). */
|
||||
async retryList(stored) {
|
||||
const { statusContext } = this.config.landing;
|
||||
const kept = [];
|
||||
for (const w of stored) {
|
||||
@@ -1900,7 +1919,7 @@ class Tick {
|
||||
);
|
||||
}
|
||||
}
|
||||
this.setLandWithdrawals(kept);
|
||||
return kept;
|
||||
}
|
||||
|
||||
async afterLanded(f, sha) {
|
||||
@@ -3535,39 +3554,48 @@ export function failedTickRecord(error) {
|
||||
return FAILED_TICKS.get(error) ?? null;
|
||||
}
|
||||
|
||||
/** `state` with landWithdrawals = `list` (pure); an empty list leaves no field. */
|
||||
function withLandWithdrawals(state, list) {
|
||||
/** The two lists of refused withdrawals, retried alike: the main one, then its overflow. */
|
||||
const WITHDRAWAL_LISTS = Object.freeze(["landWithdrawals", "landWithdrawalsOverflow"]);
|
||||
|
||||
/** `state` with `field` = `list` (pure); an empty list leaves no field. */
|
||||
function withWithdrawalList(state, field, list) {
|
||||
const next = { ...state };
|
||||
if (list.length) next.landWithdrawals = list;
|
||||
else delete next.landWithdrawals;
|
||||
if (list.length) next[field] = list;
|
||||
else delete next[field];
|
||||
return next;
|
||||
}
|
||||
|
||||
/**
|
||||
* Store one refused withdrawal in state.landWithdrawals (pure): one entry per SHA (a new refusal
|
||||
* of a stored SHA updates `why` and counts an attempt, `since` stays), at most
|
||||
* LAND_WITHDRAWALS_LIMIT. Nothing is ever dropped to make room: when the list is full and holds
|
||||
* no entry for the SHA, `stored` is false and the state is returned unchanged (the caller pauses
|
||||
* the base, pauseForUnrecordedWithdrawals).
|
||||
* Store one refused withdrawal (pure), one entry per SHA across both lists: a SHA already stored
|
||||
* in either list gets `why` updated and one more attempt (`since` stays); a new one goes to
|
||||
* state.landWithdrawals while it has room (LAND_WITHDRAWALS_LIMIT), else to
|
||||
* state.landWithdrawalsOverflow (same shape, same limit). Nothing is ever dropped to make room.
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string, candidateSha: string, why: string, at: string}} refusal
|
||||
* @returns {{state: QueueState, stored: boolean}}
|
||||
* @returns {{state: QueueState, list: string|null}} the list that holds it now, or null when both
|
||||
* are full (the state is then unchanged and the caller names the commit,
|
||||
* recordFullWithdrawals)
|
||||
*/
|
||||
export function storeWithdrawalIn(state, refusal) {
|
||||
const list = [...(state.landWithdrawals ?? [])];
|
||||
const i = list.findIndex((w) => w.sha === refusal.sha);
|
||||
if (i >= 0) {
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
const list = [...(state[field] ?? [])];
|
||||
const i = list.findIndex((w) => w.sha === refusal.sha);
|
||||
if (i < 0) continue;
|
||||
list[i] = { ...list[i], why: refusal.why, attempts: list[i].attempts + 1 };
|
||||
return { state: withLandWithdrawals(state, list), stored: true };
|
||||
return { state: withWithdrawalList(state, field, list), list: field };
|
||||
}
|
||||
if (list.length >= LAND_WITHDRAWALS_LIMIT) return { state, stored: false };
|
||||
const { sha, batchId, candidateSha, why, at } = refusal;
|
||||
list.push({ sha, batchId, candidateSha, why, since: at, attempts: 1 });
|
||||
return { state: withLandWithdrawals(state, list), stored: true };
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
const list = state[field] ?? [];
|
||||
if (list.length >= LAND_WITHDRAWALS_LIMIT) continue;
|
||||
const entry = { sha, batchId, candidateSha, why, since: at, attempts: 1 };
|
||||
return { state: withWithdrawalList(state, field, [...list, entry]), list: field };
|
||||
}
|
||||
return { state, list: null };
|
||||
}
|
||||
|
||||
/** Start of the pause detail of withdrawals a full landWithdrawals list could not keep. */
|
||||
export const WITHDRAWALS_FULL_DETAIL = "land-status withdrawals not recorded";
|
||||
/** Start of the pause detail and of the `error` event of a full landWithdrawals list. */
|
||||
export const WITHDRAWALS_FULL_DETAIL = "landWithdrawals is full";
|
||||
|
||||
/**
|
||||
* Landings of `f` that ended without a verdict, as stored: `landAttempts` at any stage (the
|
||||
@@ -3595,65 +3623,85 @@ function leaveLandingIn(state, { keepCount = false } = {}) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Pause the base with reason `error` for refused withdrawals that the full landWithdrawals list
|
||||
* could not keep (pure). Nothing is dropped from the list; each commit is named in full in the
|
||||
* pause detail, which says what to check, and one `error` event carries the same detail. A pause
|
||||
* of this kind already in place gets the new commits appended; any other pause is replaced and
|
||||
* quoted at the end of the detail (a base-red pause would lift by itself when the tip moves and
|
||||
* take the names with it). A batch at the landing stage goes back to testing: a paused base holds
|
||||
* no blessed batch.
|
||||
* Record (pure) the refusals of a tick that found state.landWithdrawals full: those it kept in
|
||||
* state.landWithdrawalsOverflow (`list` "landWithdrawalsOverflow") and those it could keep nowhere
|
||||
* (`list` null, both lists full). A full list means GitHub refuses the App's status writes, so the
|
||||
* queue must not go on blessing:
|
||||
* - a base that is not paused is paused with reason `error`; the detail says how many entries
|
||||
* each list holds and names in full each commit kept nowhere (the only record of it, with the
|
||||
* event below); the batch in flight leaves the landing stage (its count reset, as for any
|
||||
* pause);
|
||||
* - a base already paused keeps its pause untouched (reason, detail, since), except a pause of
|
||||
* this kind, which gets the commits kept nowhere appended;
|
||||
* - in every case one `error` event says the same, its `sha` the first commit kept nowhere, else
|
||||
* the first kept in the overflow.
|
||||
* @param {QueueState} state
|
||||
* @param {{sha: string, batchId: string}[]} unrecorded
|
||||
* @param {{sha: string, batchId: string, list: string|null}[]} refusals
|
||||
* @param {{at: string, url?: string, statusContext: string}} options
|
||||
*/
|
||||
export function pauseForUnrecordedWithdrawals(state, unrecorded, { at, url, statusContext }) {
|
||||
export function recordFullWithdrawals(state, refusals, { at, url, statusContext }) {
|
||||
const bySha = new Map();
|
||||
for (const r of refusals) bySha.set(r.sha, r);
|
||||
const all = [...bySha.values()];
|
||||
if (!all.length) return state;
|
||||
const nowhere = all.filter((r) => r.list === null);
|
||||
const named = (list) => list.map((r) => `${r.sha} (${r.batchId})`).join(", ");
|
||||
const counts =
|
||||
`${(state.landWithdrawals ?? []).length} of ${LAND_WITHDRAWALS_LIMIT}; ` +
|
||||
`landWithdrawalsOverflow holds ${(state.landWithdrawalsOverflow ?? []).length} of ` +
|
||||
`${LAND_WITHDRAWALS_LIMIT}`;
|
||||
const lost = nowhere.length
|
||||
? `. Kept nowhere (both lists full), the only record of them: ${named(nowhere)}; check ` +
|
||||
`their ${statusContext} by hand (docs/ops/OMNI_QUEUE.md, Incidents)`
|
||||
: "";
|
||||
const paused = state.paused;
|
||||
const listed =
|
||||
const ours =
|
||||
paused?.reason === "error" && String(paused.detail ?? "").startsWith(WITHDRAWALS_FULL_DETAIL);
|
||||
const named = new Set();
|
||||
const fresh = [];
|
||||
for (const u of unrecorded) {
|
||||
if (named.has(u.sha) || (listed && paused.detail.includes(u.sha))) continue;
|
||||
named.add(u.sha);
|
||||
fresh.push(u);
|
||||
let next = state;
|
||||
if (!paused) {
|
||||
const detail =
|
||||
`${WITHDRAWALS_FULL_DETAIL} (${counts}): GitHub refused the App's ${statusContext} ` +
|
||||
"failure again; every refused withdrawal is kept and retried on every tick. Find out why " +
|
||||
"GitHub refuses the App's status writes, wait until both lists are empty, then resume" +
|
||||
lost;
|
||||
next = leaveLandingIn({
|
||||
...state,
|
||||
paused: { reason: "error", since: at, baseSha: null, detail, issue: null },
|
||||
});
|
||||
} else if (ours) {
|
||||
const fresh = nowhere.filter((r) => !paused.detail.includes(r.sha));
|
||||
if (fresh.length) {
|
||||
const detail = `${paused.detail}; also kept nowhere: ${named(fresh)}`;
|
||||
next = { ...state, paused: { ...paused, detail } };
|
||||
}
|
||||
}
|
||||
if (!fresh.length) return leaveLandingIn(state);
|
||||
const commits = fresh.map((u) => `${u.sha} (${u.batchId})`).join(", ");
|
||||
const detail = listed
|
||||
? `${paused.detail}; also not recorded: ${commits}`
|
||||
: `${WITHDRAWALS_FULL_DETAIL}: landWithdrawals already holds ${LAND_WITHDRAWALS_LIMIT} ` +
|
||||
`entries, so the refused withdrawal of ${statusContext} on ${commits} is stored nowhere ` +
|
||||
"(nothing was dropped). Check why GitHub refuses the App's status writes, and the land " +
|
||||
"status of each commit named here (docs/ops/OMNI_QUEUE.md, Incidents); resume once " +
|
||||
"landWithdrawals has room" +
|
||||
(paused ? ` (was paused: ${paused.reason}: ${String(paused.detail).slice(0, 200)})` : "");
|
||||
const next = {
|
||||
...state,
|
||||
paused: {
|
||||
reason: "error",
|
||||
since: listed ? paused.since : at,
|
||||
baseSha: null,
|
||||
detail,
|
||||
issue: null,
|
||||
},
|
||||
};
|
||||
const event = stripUndefined({
|
||||
at,
|
||||
type: "error",
|
||||
batchId: fresh[0].batchId,
|
||||
sha: fresh[0].sha,
|
||||
detail: detail.slice(0, 500),
|
||||
url,
|
||||
});
|
||||
return leaveLandingIn(appendEvent(next, event));
|
||||
const overflowed = all.filter((r) => r.list !== null).map((r) => head12(r.sha));
|
||||
const detail =
|
||||
`${WITHDRAWALS_FULL_DETAIL} (${counts})` +
|
||||
(overflowed.length
|
||||
? `: refused withdrawals kept in the overflow: ${overflowed.join(", ")}`
|
||||
: "") +
|
||||
(nowhere.length ? `; kept nowhere: ${named(nowhere)}` : "") +
|
||||
(paused && !ours ? `; the base stays paused (${paused.reason})` : "");
|
||||
const first = nowhere[0] ?? all[0];
|
||||
return appendEvent(
|
||||
next,
|
||||
stripUndefined({
|
||||
at,
|
||||
type: "error",
|
||||
batchId: first.batchId,
|
||||
sha: first.sha,
|
||||
detail: detail.slice(0, 500),
|
||||
url,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply to the stored state what a tick that threw left for it (failedTickRecord), pure: drop
|
||||
* the stored withdrawals it settled, take the batch it withdrew off the landing stage when the
|
||||
* stored batch is still that one at that stage (its landing count kept), store the withdrawals
|
||||
* GitHub refused (and pause the base for any the full list cannot keep,
|
||||
* pauseForUnrecordedWithdrawals). A null record
|
||||
* GitHub refused (and record a full list, recordFullWithdrawals). A null record
|
||||
* returns `state` unchanged.
|
||||
* @param {QueueState} state
|
||||
* @param {ReturnType<typeof failedTickRecord>} record
|
||||
@@ -3663,11 +3711,14 @@ export function amendFailedTick(state, record, options) {
|
||||
if (!record) return state;
|
||||
let next = state;
|
||||
const retired = new Set(record.retired ?? []);
|
||||
if (retired.size && next.landWithdrawals) {
|
||||
next = withLandWithdrawals(
|
||||
next,
|
||||
next.landWithdrawals.filter((w) => !retired.has(w.sha))
|
||||
);
|
||||
for (const field of WITHDRAWAL_LISTS) {
|
||||
if (retired.size && next[field]) {
|
||||
next = withWithdrawalList(
|
||||
next,
|
||||
field,
|
||||
next[field].filter((w) => !retired.has(w.sha))
|
||||
);
|
||||
}
|
||||
}
|
||||
const leave = record.leaveLanding;
|
||||
const f = next.inFlight;
|
||||
@@ -3679,13 +3730,13 @@ export function amendFailedTick(state, record, options) {
|
||||
) {
|
||||
next = leaveLandingIn(next, { keepCount: true });
|
||||
}
|
||||
const unrecorded = [];
|
||||
const full = [];
|
||||
for (const refusal of record.withdrawals ?? []) {
|
||||
const result = storeWithdrawalIn(next, refusal);
|
||||
next = result.state;
|
||||
if (!result.stored) unrecorded.push(refusal);
|
||||
if (result.list !== "landWithdrawals") full.push({ ...refusal, list: result.list });
|
||||
}
|
||||
return unrecorded.length ? pauseForUnrecordedWithdrawals(next, unrecorded, options) : next;
|
||||
return full.length ? recordFullWithdrawals(next, full, options) : next;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -365,6 +365,66 @@ test("load: a malformed landWithdrawals is refused, and never saved", async () =
|
||||
}
|
||||
});
|
||||
|
||||
test("load: landWithdrawalsOverflow is absent in older documents, and accepted up to LAND_WITHDRAWALS_LIMIT entries, one per SHA across both lists", async () => {
|
||||
const sha = (n) => n.toString(16).padStart(40, "0");
|
||||
const main = Array.from({ length: 20 }, (_, i) => withdrawalEntry(i, { sha: sha(i) }));
|
||||
const overflow = Array.from({ length: 20 }, (_, i) => withdrawalEntry(i, { sha: sha(100 + i) }));
|
||||
for (const state of [
|
||||
withWithdrawals(main),
|
||||
{ ...withWithdrawals(main), landWithdrawalsOverflow: overflow },
|
||||
{ ...initialState(BASE), landWithdrawalsOverflow: [overflow[0]] },
|
||||
]) {
|
||||
const { gh, store } = setup();
|
||||
gh.contents.set(KEY, { content: JSON.stringify(state), sha: "s1" });
|
||||
const loaded = (await store.load(BASE)).state;
|
||||
assert.deepEqual(loaded.landWithdrawalsOverflow, state.landWithdrawalsOverflow);
|
||||
assert.deepEqual(loaded.landWithdrawals, state.landWithdrawals);
|
||||
}
|
||||
});
|
||||
|
||||
test("load: a malformed landWithdrawalsOverflow is refused, and never saved", async () => {
|
||||
const sha = (n) => n.toString(16).padStart(40, "0");
|
||||
const cases = [
|
||||
["not an array", { sha: sha(1) }, /\(landWithdrawalsOverflow\)/],
|
||||
[
|
||||
"more than 20",
|
||||
Array.from({ length: 21 }, (_, i) => withdrawalEntry(i, { sha: sha(i) })),
|
||||
/\(landWithdrawalsOverflow\)/,
|
||||
],
|
||||
["a short sha", [withdrawalEntry(1, { sha: "abc" })], /landWithdrawalsOverflow\[0\]\.sha/],
|
||||
[
|
||||
"zero attempts",
|
||||
[withdrawalEntry(1, { attempts: 0 })],
|
||||
/landWithdrawalsOverflow\[0\]\.attempts/,
|
||||
],
|
||||
];
|
||||
for (const [what, landWithdrawalsOverflow, pattern] of cases) {
|
||||
const { gh, store } = setup();
|
||||
const state = { ...initialState(BASE), landWithdrawalsOverflow };
|
||||
gh.contents.set(KEY, { content: JSON.stringify(state), sha: "s1" });
|
||||
await assert.rejects(store.load(BASE), pattern, what);
|
||||
await assert.rejects(store.save(BASE, state, null, "bad"), StateError, what);
|
||||
assert.deepEqual(gh.writes, [], what);
|
||||
}
|
||||
// one per SHA across both lists
|
||||
const { gh, store } = setup();
|
||||
const both = {
|
||||
...withWithdrawals([withdrawalEntry(1)]),
|
||||
landWithdrawalsOverflow: [withdrawalEntry(2, { sha: withdrawalEntry(1).sha })],
|
||||
};
|
||||
gh.contents.set(KEY, { content: JSON.stringify(both), sha: "s1" });
|
||||
await assert.rejects(store.load(BASE), /landWithdrawalsOverflow\[0\]\.sha/);
|
||||
});
|
||||
|
||||
test("a top-level field the validator does not know is ignored: how code before landWithdrawalsOverflow loads a state that has it", async () => {
|
||||
// The validators of 05b9bd4 and of release/v3.8.54 check known fields only (verified for this
|
||||
// field when it was added); this pins that rule for the current one.
|
||||
const { gh, store } = setup();
|
||||
const state = { ...initialState(BASE), aFieldOfLaterCode: [{ anything: true }] };
|
||||
gh.contents.set(KEY, { content: JSON.stringify(state), sha: "s1" });
|
||||
assert.deepEqual((await store.load(BASE)).state.aFieldOfLaterCode, [{ anything: true }]);
|
||||
});
|
||||
|
||||
test("save refuses an invalid state or a base mismatch without writing", async () => {
|
||||
const { gh, store } = setup();
|
||||
const invalid = { ...initialState(BASE), inFlight: { batchId: "../x" } };
|
||||
|
||||
@@ -43,6 +43,7 @@ import {
|
||||
readOnlyGitHub,
|
||||
recordTickFailure,
|
||||
runTick,
|
||||
WITHDRAWALS_FULL_DETAIL,
|
||||
} from "../../../scripts/merge-queue/tick.mjs";
|
||||
import { WATCHDOG_LIMIT_MINUTES } from "../../../scripts/merge-queue/watchdog.mjs";
|
||||
import { startFakeGitHub } from "./helpers/fake-github.mjs";
|
||||
@@ -7318,7 +7319,11 @@ async function fullListThenRefusedLanding(h) {
|
||||
return { f, old };
|
||||
}
|
||||
|
||||
test("M4: twenty stored withdrawals and a twenty-first refusal: the base pauses, nothing is dropped, the detail names the SHA", async () => {
|
||||
/** The entries of state.landWithdrawalsOverflow as [sha, batchId, attempts]. */
|
||||
const overflowOf = (state) =>
|
||||
(state.landWithdrawalsOverflow ?? []).map((w) => [w.sha, w.batchId, w.attempts]);
|
||||
|
||||
test("M4: twenty stored withdrawals and a twenty-first refusal: the base pauses, nothing is dropped, the refusal is kept in the overflow", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f, old } = await fullListThenRefusedLanding(h);
|
||||
refuseWithdrawals(h, () => true);
|
||||
@@ -7334,13 +7339,21 @@ test("M4: twenty stored withdrawals and a twenty-first refusal: the base pauses,
|
||||
Array(20).fill(2),
|
||||
"each retried (and refused) once more"
|
||||
);
|
||||
assert.deepEqual(
|
||||
overflowOf(state),
|
||||
[[f.candidateSha, f.batchId, 2]],
|
||||
"the refusal that did not fit is stored in the overflow (path + end-of-tick pass)"
|
||||
);
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.ok(state.paused.detail.includes(f.candidateSha), "the commit not recorded, in full");
|
||||
assert.ok(state.paused.detail.includes(f.batchId));
|
||||
assert.match(state.paused.detail, /landWithdrawals/);
|
||||
assert.ok(
|
||||
state.paused.detail.startsWith(`${WITHDRAWALS_FULL_DETAIL} (20 of 20; `),
|
||||
state.paused.detail
|
||||
);
|
||||
assert.match(state.paused.detail, /landWithdrawalsOverflow holds 1 of 20/);
|
||||
assert.match(state.paused.detail, /resume/);
|
||||
assert.ok(!state.paused.detail.includes(f.candidateSha), "stored, so not named as unrecorded");
|
||||
assert.equal(state.history.at(-1).type, "error");
|
||||
assert.ok(state.history.at(-1).detail.includes(f.candidateSha));
|
||||
assert.equal(state.history.at(-1).sha, f.candidateSha);
|
||||
assert.ok(
|
||||
!state.history.some((e) => /stored withdrawal of .* dropped/.test(e.detail ?? "")),
|
||||
"no entry dropped, so no dropped event"
|
||||
@@ -7348,7 +7361,7 @@ test("M4: twenty stored withdrawals and a twenty-first refusal: the base pauses,
|
||||
assert.notEqual(state.inFlight?.stage, "landing", "a paused base holds no blessed batch");
|
||||
});
|
||||
|
||||
test("M4: while paused for a full list the stored withdrawals are still retried; a resume with room continues", async () => {
|
||||
test("M4: while paused for a full list both lists are still retried; a resume with room continues", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f, old } = await fullListThenRefusedLanding(h);
|
||||
refuseWithdrawals(h, () => true);
|
||||
@@ -7363,12 +7376,10 @@ test("M4: while paused for a full list the stored withdrawals are still retried;
|
||||
await h.tick();
|
||||
let state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error", "still paused: the operator resumes");
|
||||
assert.equal(
|
||||
"landWithdrawals" in state,
|
||||
false,
|
||||
"every stored withdrawal was retried and written"
|
||||
);
|
||||
assert.equal("landWithdrawals" in state, false, "every stored withdrawal was written");
|
||||
assert.equal("landWithdrawalsOverflow" in state, false, "the overflow too");
|
||||
for (const sha of old) assert.equal(newestLand(h, sha), "failure");
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure");
|
||||
assert.deepEqual(state.inFlight, recomposed, "the batch in flight is left as is");
|
||||
|
||||
await h.tick({ action: "resume" });
|
||||
@@ -7381,7 +7392,56 @@ test("M4: while paused for a full list the stored withdrawals are still retried;
|
||||
assert.equal(state.paused, null);
|
||||
});
|
||||
|
||||
test("M4: the failure path pauses too when a refused withdrawal finds the list full", async () => {
|
||||
test("M4 (review probe 3): after the drain and the resume, the twenty-first commit is withdrawn too and both lists are empty", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f } = await fullListThenRefusedLanding(h);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
h.gh.writeError = null;
|
||||
for (let i = 0; i < 3; i++) {
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
}
|
||||
await h.tick({ action: "resume" });
|
||||
for (let i = 0; i < 3; i++) {
|
||||
h.clock.advance(10);
|
||||
await h.tick();
|
||||
}
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused, null);
|
||||
assert.equal(newestLand(h, f.candidateSha), "failure", "never left with the App's success");
|
||||
assert.equal("landWithdrawals" in state, false);
|
||||
assert.equal("landWithdrawalsOverflow" in state, false);
|
||||
});
|
||||
|
||||
test("M4: both lists full: the commit that fits nowhere is named in the pause detail and in the event's sha", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const { f, old } = await fullListThenRefusedLanding(h);
|
||||
const spill = Array.from({ length: 20 }, (_, i) => fakeSha(`spill-${i}`));
|
||||
for (const sha of spill) setLand(h, sha, "success");
|
||||
const { state: stored, version } = await h.store.load(BASE);
|
||||
stored.landWithdrawalsOverflow = spill.map((sha) => storedWithdrawal(sha));
|
||||
assert.equal((await h.store.save(BASE, stored, version, "seed: overflow")).ok, true);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick();
|
||||
const state = await h.state();
|
||||
assert.deepEqual(
|
||||
state.landWithdrawals.map((w) => w.sha),
|
||||
old
|
||||
);
|
||||
assert.deepEqual(
|
||||
state.landWithdrawalsOverflow.map((w) => w.sha),
|
||||
spill
|
||||
);
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.match(state.paused.detail, /landWithdrawalsOverflow holds 20 of 20/);
|
||||
assert.ok(state.paused.detail.includes(`${f.candidateSha} (${f.batchId})`), "named in full");
|
||||
const event = state.history.findLast((e) => e.type === "error");
|
||||
assert.equal(event.sha, f.candidateSha);
|
||||
assert.ok(event.detail.includes(f.candidateSha));
|
||||
});
|
||||
|
||||
test("M4: the failure path pauses too when a refused withdrawal finds the list full; the refusal is in the overflow", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
@@ -7395,11 +7455,44 @@ test("M4: the failure path pauses too when a refused withdrawal finds the list f
|
||||
await recordedFailingTick(h, /not available from origin/);
|
||||
const state = await h.state();
|
||||
assert.equal(state.paused?.reason, "error");
|
||||
assert.ok(state.paused.detail.includes(f.candidateSha));
|
||||
assert.ok(state.paused.detail.startsWith(WITHDRAWALS_FULL_DETAIL), state.paused.detail);
|
||||
assert.equal(state.landWithdrawals.length, 20);
|
||||
assert.deepEqual(overflowOf(state), [[f.candidateSha, f.batchId, 1]]);
|
||||
assert.notEqual(state.inFlight?.stage, "landing", "a paused base holds no blessed batch");
|
||||
});
|
||||
|
||||
test("M4 (review probe 6): a base already paused by hand keeps its pause when the list is full; one resume lifts that pause", async () => {
|
||||
const h = createHarness({ mode: "live", actor: "owner" });
|
||||
const f = await greenBatch(h, [1]);
|
||||
const old = Array.from({ length: 20 }, (_, i) => fakeSha(`old-${i}`));
|
||||
for (const sha of old) setLand(h, sha, "success");
|
||||
await storeWithdrawals(
|
||||
h,
|
||||
old.map((sha) => storedWithdrawal(sha))
|
||||
);
|
||||
await diedAfterBlessing(h, f, [f.candidateSha]);
|
||||
refuseWithdrawals(h, () => true);
|
||||
await h.tick({ action: "pause", reason: "before the mode change" });
|
||||
let state = await h.state();
|
||||
assert.equal(state.paused?.reason, "manual", "the operator's pause is kept");
|
||||
assert.equal(state.paused.detail, "before the mode change", "untouched");
|
||||
assert.deepEqual(overflowOf(state), [[f.candidateSha, f.batchId, 1]]);
|
||||
const full = state.history.findLast((e) => e.type === "error");
|
||||
assert.ok(full.detail.startsWith(WITHDRAWALS_FULL_DETAIL), full.detail);
|
||||
assert.equal(full.sha, f.candidateSha);
|
||||
|
||||
h.gh.writeError = null;
|
||||
await h.tick({ action: "resume" });
|
||||
state = await h.state();
|
||||
assert.equal(state.paused, null);
|
||||
assert.equal(
|
||||
state.history.findLast((e) => e.type === "resumed").detail,
|
||||
"manual resume (was manual)",
|
||||
"the pause lifted is the operator's own"
|
||||
);
|
||||
assert.equal("landWithdrawalsOverflow" in state, false, "the overflow was retried and written");
|
||||
});
|
||||
|
||||
// ── a batch that did land, met by a pause or a shadow tick ──
|
||||
|
||||
/** The landing push went through and the tick lost its save: base = candidate, PRs merged. */
|
||||
|
||||
Reference in New Issue
Block a user