* fix(scripts): restore derived files byte for byte in the strip pass
The strip pass read merge-base and head contents through a helper that
trims its output, so every restored file lost its final newline. The
"restored" blob still differed from the merge base and the PR kept
conflicting on it (stryker.conf.json in the first batch).
Read blobs with a dedicated readBlob() that keeps the exact bytes, and
add a regression test that builds the restore commit from a real git
repository and compares blob ids with the merge base.
* fix(scripts): restore a claim with the base's own text in the strip pass
restoreClaims wrote the parsed base value back as an ASCII number. A
translated mirror that writes the count in native digits (Persian
README: "۲۳۳ مهاجرت") got "233" instead, so the line still differed from
the merge base and the PR kept conflicting on that mirror.
Use the base's exact substring for the claim span. Regression test with
a Persian-digit claim fails on the old code and passes now.
The status rule validator accepted do_not_enforce_on_create: true and refused
a strict policy only when it was exactly true, so a non-boolean passed. Both
flags must now be exactly false. Tests cover the three refusals no test
exercised: a status rule without parameters, an unexpected parameter, and a
template that pins an integration_id itself. The hint printed when another
ruleset would reject the landing push now says what the runbook says: remove
those rulesets, the status door replaces them.
createGit spread process.env into every git child, so OMNI_QUEUE_TOKEN and
OMNI_QUEUE_LAND_TOKEN reached every git process, although credentials travel
through GIT_CONFIG_* already. Both are now removed from the base environment
and from any per-call environment.
The candidate verdict ignores statuses by the omni-queue/ prefix: a land
status outside it would let a withdrawn (failure) status turn a candidate
red, and one equal to an approval context would make a label approval a
blessing. config.mjs now refuses both; the Zod mirror and the differential
table carry the same rule.
A shadow bisection composes the next segment on the would-land commit, but
that commit was never recorded green, so a red segment stacked on it paid a
full base check of a commit that had just passed. A would-land inside a
bisection now records its candidate green, as live mode records the landed
tip.
Withdrawing the land status was all-or-nothing and could be skipped: a
refused write stopped revokeLanding and skipped the pause, and a tick that
died after blessing left no record, so the next tick recomposed with the
status still in success.
revokeLanding now attempts every commit, never throws, and records a failed
withdrawal in an error event, a note and the rethrown error. The in-flight
stage "landing" is saved before the App blesses (the gate: only a batch at
that stage pays a status read), and at the end of every live tick a batch
stored or marked at that stage that was neither landed nor kept there has
each newest land status read and turned into failure where it is success. A
manual pause puts such a batch back to the testing stage, so its status is
withdrawn until it lands after the resume.
The tick blessed and landed whatever the stored in-flight record named: a
record with no PR, or a candidate with an extra commit, got the land status.
Before the would-land and the bless, land.verifyComposition now requires a
non-empty batch, the base as an ancestor of the candidate, a first-parent
chain made only of queue merge commits that the stored PRs consume in order
at their heads, and stacked PRs (no merge commit of their own) exactly where
composeCandidate puts them; squash-api also checks the stored tree. A
mismatch blesses and lands nothing, records no would-land, and pauses the
queue with reason "error".
A re-run keeps the original dispatcher's inputs (sha, reason) and changes only
github.triggering_actor, so an approver re-running a non-approver's failed bless
used to write the land status. The tick now refuses a bless unless GITHUB_ACTOR
and GITHUB_TRIGGERING_ACTOR are both approvers, refuses when either is missing,
and names both in the status and the event when they differ. The workflow passes
github.actor next to github.triggering_actor.
A hang signature recorded on a base check now matches only a hang of
the candidate on the same check: a base that times out a job never
excuses an ordinary keyless failure of that job (base-red pause before,
pr-red now).
The hang path requires every run of the candidate to have been only job
timeouts. The in-flight retry carries timedOutRuns while that holds;
validateState accepts it as a count, and a document written without it
keeps the old behaviour (pause with reason infra). The hang-suspected
detail and the culprit comment state the real number of timed-out runs.
The culprit comment says the base passes the job only on a green base
verdict, and says nothing about the base without a verdict. The fake
classifier follows the real precedence for timeout and hang verdicts.
A job that hits its time limit on every attempt is a hanging test, not
the machine. When the infra retries are used up and every failure is
hang-shaped (timed_out, or cancelled with the maximum-execution-time
annotation; never runner loss, startup_failure or a plain cancel), the
tick records hang-suspected and classifies the hang signatures like any
red batch: base check when the base has no verdict, base-red pause when
the bare base fails the same job, otherwise bisection with hang: true,
whose timed-out segments are not retried. The culprit comment names the
job that timed out.
A base check that never produces a verdict (no CI run, or no result
after checksTimeoutMinutes) and has no retry left no longer pauses the
queue: it is recorded as a timeout verdict and the parked batch is
settled (pr-red, the check cannot be redone for the same SHA). The
verdict lookup for a later red batch accepts timeout, so no base check
is started again for that SHA.
A job timeout on the bare base is recorded as a hang signature of the
base check even next to other failures, so a batch that hangs on the
same job is matched to the base instead of being blamed.
Both jobs were failing gates on time, with every test green.
Unit shard, 25 -> 35 min: over 72 green shard jobs on 2026-11-14 the mean
was 19 min and the slowest 24, and 19 of them (26%) ended within 3 min of
the limit. A shard whose steps had all passed was cancelled at 25 min
(run 43651743993) and failed Gate / CI until a re-run.
Coverage, 45 -> 60 min: two pull requests died at exactly 45 min on the
same day (runs 43619286507 and 43625515181); the re-run that passed took
39m45s.
The Release-Green observer comment that quotes the shard ceiling follows.
The LEGACY_MIGRATION_MAX comment said 241_seen_models.sql came from
#18359. It came from #16864, renumbered from 240 to 241 in merge train 2;
#18359 was stacked on #18353 and carries no migration of its own. The
commit that raised the freeze to 241 (d0ec9e8b) has the same slip in its
message.
Follow-up to the review of the pause/resume confirmation. `resume` reported success as soon
as the base was unpaused, also when it already was, while its dispatch could still be
pending in the tick's concurrency group and later clear a pause recorded in between (a
safety pause). Now the CLI sends nothing for a base that is not paused, and a resume carries
the pause it saw (`expect-pause:<since>`); the tick clears only that pause. A second
dispatch of the same request is therefore harmless, which also covers a retry made after
another request's run was mistaken for ours. The existing test that pinned an empty resume
reason now pins the new format.
The gate pins its comparison to the event's commits. A push that creates a branch has no
`before`, and every omni-queue candidate is such a push, so the gate ended INCOMPLETE
("Missing immutable event base") and the candidate went red: found by the real-repository
dry run of the candidate profile (run 43614500902). On refs/heads/omni-queue/** the base is
now the one the candidate names itself (Omni-Queue-Base, or the commit itself for a
base check), through candidate-base.resolveCandidateBase, which fails closed when the ref
does not name the commit. Every other push keeps event.before.
Shadow mode writes nothing on a pull request, but the approval job still posted its
"not an approver" comment. In shadow a label from a non-approver is now only logged.
The main() test that pins that comment hands main a live config (setup only; the
assertions are unchanged). The protected-surface comment and two doc comments now say
"protected surfaces": the list also covers workflows and the queue's own code and config.
Every tick called store.save, and a save stamps `updatedAt`, so the
document always differed: one commit on `omni-queue-state` per tick, at
least 288 a day per base from the cron alone, although most ticks change
nothing (idle queue, candidate checks still pending).
The tick now compares the document it loaded with the one it ends with,
`updatedAt` aside, and skips the save when they are equal. An unchanged
state is still written as a heartbeat at most once per
STATE_HEARTBEAT_MINUTES (60, exported from tick.mjs), so the age of
`updatedAt` keeps telling an idle queue from a dead one. A missing
document is always written, and so is one whose `updatedAt` cannot be
read or lies in the future.
Why a skipped save is safe:
- Compare-and-swap. The CAS exists so that a write never overwrites a
concurrent one. A tick with nothing to write cannot lose an update, and
the version it loaded is thrown away with it; the next tick loads its
own. A tick that does change the state saves with the version it
loaded, exactly as before, and still fails on a conflict.
- Side effects. What an unchanged tick does on GitHub (dashboard upsert,
eligibility comments and label removals, deletion of stray candidate
refs) already ran before the save in every tick and is not recorded in
the state. A save conflict never undid it; it only made main() redo the
tick. With the skip, a concurrent change is picked up by the next tick.
- Ref cleanup. cleanupRefs runs before the save and keeps both the loaded
in-flight refs and the refs of the new state. deleteSuperseded runs
after it and deletes the loaded refs that the new state no longer
names. With an identical document those two sets are equal, so it
deletes nothing: no ref is removed on the strength of a save that did
not happen.
The dashboard is still rendered and upserted on every tick (it already
PATCHes only when its body changes). No existing assertion changed.
A push to omni-queue/** has no github.event.pull_request, so the gates that
are base-relative (or advisory) on a pull request ran a candidate in their
absolute / strict mode. A base that drifted mid-cycle would red every
candidate and the queue would blame innocent pull requests. Measured on the
release tip: the strict i18n drift check exits 1 (33 sources, 66 mirrors).
On a candidate these now compare with needs.changes.outputs.candidateBase:
- ci.yml lint: boundary ratchet (--base-ref).
- quality.yml fast-gates: file-size, complexity-ratchets, dead-code, boundary
and the TS7 diagnostics ratchet (which was skipped without a base).
- ci.yml docs-sync-strict: the i18n docs drift takes the pull-request path
(base checkout of the candidate base + advisory comparison); the strict
step no longer runs on a candidate.
Pushes to main, merge_group and workflow_dispatch evaluate to the same
values as before; pull requests keep github.event.pull_request.base.sha.
newCodeMode.resolveMergeBase: on a merge-commit HEAD it compares with HEAD^1
(the live base tip of refs/pull/N/merge). A batch candidate is the base plus
one merge commit per pull request, so that rule judged only the last pull
request of the batch. The two new-code gates now pass the pushed ref, and a
refs/heads/omni-queue/** ref compares with the given base. Every other ref
keeps the first-parent rule. GITHUB_REF joins the runner-owned variables of
check-env-doc-sync (the gates read it; it is not OmniRoute configuration).
Existing assertions changed on purpose (same strictness, new expression):
- tests/unit/build/check-boundary-ratchet.test.ts: the two exact matches of
the PR_BASE_SHA binding (ci.yml lint, quality.yml fast-gates) now expect
"github.event.pull_request.base.sha || needs.changes.outputs.candidateBase".
- tests/unit/quality-rail-gate-membership.test.ts: the PR_BASE_SHA regex of
the TS7 ratchet expects the same expression.
Shadow mode is meant to observe while Mergify keeps landing the same PRs.
It still acted on them:
- the tick removed the queue label and commented when it isolated a
culprit, when a PR conflicted with the base, when its head was already
in the base, and on every eligibility action (head changed after the
label, conflicting, not an approver, protected surface, frozen base);
- the approval job wrote a `failure` status "approval withdrawn" on every
`unlabeled`, a red status on a healthy PR that the other merge path may
refuse to merge;
- a PR of the in-flight batch merged by Mergify during the candidate CI
made the green batch be dropped as "already merged" and recomposed, so
shadow rarely recorded `would-land`.
In shadow mode only (live mode is unchanged):
- tick.mjs writes to a pull request through two methods, removeLabel and
the new commentPr, and both are no-ops in shadow. Comments on issues the
queue owns (base-red notice, divergence, dashboard) keep going through
comment().
- What live mode would have done is recorded instead. A culprit keeps the
`culprit-found` event, with "shadow mode: label kept, no comment". A
base conflict, a head already in the base and an eligibility label
removal become a new `would-dequeue` event (PR, head, reason).
- A culprit, a base conflict and an already-in-base head are remembered
per head SHA in state.shadow.processed, like a validated head, so they
are not batched again (eligibility reason `shadow-processed`). A PR the
eligibility rules reject is recorded once per head and not marked: it
stays out by its own reason, and a new approval of that head can still
queue it.
- A PR that another path merged at the tested head during the candidate
CI no longer discards a green batch: `would-land` is recorded with
"merged meanwhile by another path: #N". A PR merged at another head, or
dropped for any other reason, is handled as before.
- approve.mjs writes nothing for `unlabeled`. This is safe: checked in
eligibility.approvalFor, the older `success` never counts by itself.
Without the label the PR is `no-label`; a label put back by a
non-approver is `label-not-by-approver`; after an approver's new label
only a status created no earlier than 60 s before that label approves
the head. A characterization test pins this.
Not changed: the approval job still answers a non-approver's label with
one comment in shadow mode (the runbook says so).
Existing assertions changed on purpose:
- tick.test.mjs "shadow bisection composes later segments…": the culprit
#3 now keeps its label (was `[]`) and is in shadow.processed (keys were
1, 2, 4); the test also asserts the culprit-found event and that no
label removal or comment was written.
Existing tests whose setup (not assertions) changed, because they ran the
committed shadow config while pinning live behaviour:
- approve.test.mjs uses the committed config with mode "live";
- tick.test.mjs "main: the client counts only the App's own comment
markers" passes a live config through OMNI_QUEUE_CONFIG.
The candidate SHA is public. Anybody can publish that commit in a fork, or
on a branch of this repository, and open a pull request whose head is the
candidate. The PR's CI then reports check runs on the same SHA, and the
tick read them as the candidate's:
- readChecks kept every run whose check suite had no head_branch (fork
PRs) or whose head_branch was the candidate ref (a same-repository PR
from a branch with that name);
- by cancelling or re-pushing, the PR's author could make the candidate
red or "infra" (retry, then a paused base), and a later run with the
same name superseded the candidate's own result in checks.latestByName.
The verdict of a candidate ref now counts only the check runs of the check
suites of push-event workflow runs on that ref for that SHA
(GET /actions/runs?branch=<ref>&event=push&head_sha=<sha>, token already
has actions: read). Every listed run is re-checked for event, branch and
SHA. Anything else on the SHA is ignored: the base's own push, the other
ref of an infra retry, pull_request suites, workflow_dispatch runs, and
check runs of other apps (commit statuses are unchanged). Unknown fails
closed: with no ref, or no push run listed yet, no run counts, the verdict
stays pending and the existing no-CI timeout applies.
Why not strict `headBranch === ref`: checked on the release tip of the
real repository, a check suite carries the branch but not the event, so
the suites of a pull_request run and of a push run on the same branch
look alike. Each workflow run has its own check suite, and
workflow_run.check_suite_id equals check_run.check_suite.id.
github.mjs: check runs carry checkSuiteId; workflow runs carry headBranch
and checkSuiteId.
Tests: the new tick, client and end-to-end tests fail on the previous
code. Existing assertions changed, all additive, because the normalized
shapes gained fields: github.test.mjs expects checkSuiteId on
normalizeCheckRun and headBranch/checkSuiteId on listWorkflowRuns. One
tick test that injects raw check runs now attributes them to the push run
of the candidate ref in its setup; its assertions are untouched.
While this PR was open, merge train 2 merged
240_provider_node_custom_json_config.sql (#18353) and 241_seen_models.sql
(#18359) into release/v3.8.54. With the series frozen at 239, the gate
rejected both base files and the compatibility tests' synthetic 240 slot
collided with the real one, so CI went red on the merge ref.
A merged migration is never renamed, so the freeze moves to the highest
old number on the base: LEGACY_MIGRATION_MAX is now 241, and its comment
says to raise it in the same merge if another NNN_ file lands before this
PR does. The tests no longer hard-code the free slot: the timestamp
compatibility test takes the first old number above every real file, and
the numbering tests derive the top and the number above it from
LEGACY_MIGRATION_MAX. The pinned value test now pins 241. The database
guide and the changelog fragment say 001-241, and the conversion recipe
lists 237 to 241 as slots that always need a compatibility entry.
merge-ff keeps every contributor commit where a squash leaves one, so the "Commits in the
cycle" figure and the Top 25 ranking would favour a 30-commit PR over five squashed ones.
landingUnits() counts a queue PR once; rankAuthors() and renderSection() use it, and the
table names how many PRs the queue landed. Squash-only cycles render exactly as before;
the scope line keeps the literal non-merge commit count.