Compare commits

...

1 Commits

Author SHA1 Message Date
diegosouzapw
3be7e618f8 docs(ci): record what a missing App permission does to the omni-queue tick
Measured on 2026-11-17 against an installation with `actions: read`: asked
for a permission the installation does not hold, GitHub mints no token at
all (HTTP 422 for the job's whole permission set). So the tick fails at
the token step, not only the re-run of failed jobs.

The runbook said this had not been measured, described a re-run refused
with 403 as the result, and listed a revoked permission and an uninstalled
App among the causes of the full withdrawal list. Neither can happen: a
tick that gets no token stores nothing.
2026-11-17 20:03:34 -03:00

View File

@@ -756,13 +756,15 @@ The cost: a real failure is blamed later, after the rest of the CI run plus the
often turn green on their re-run are flaky tests to fix.
**The App needs Actions: Read and write.** The tick job asks for `permission-actions: write` when
it mints its token. Grant it to the App, and accept it on the installation, before the version of
the queue that re-runs failed jobs reaches the default branch (see
[Installing the GitHub App](#installing-the-github-app), step 4). A token that reaches the tick
without it gets the call refused: the `checks-red` detail ends with `; re-run refused (<status>)`
(403 is GitHub's usual answer to a missing permission), and the batch is classified in the same
tick. Whether GitHub mints a token at all for an installation that lacks a
permission the job asks for has not been measured: it may fail every tick at the token step.
it mints its token. Grant it to the App, and accept it on the installation, before a tick runs with
this version of the workflow (see [Installing the GitHub App](#installing-the-github-app), step 4).
Without it GitHub mints no token at all. Asked for a permission the installation does not hold,
`POST /app/installations/{id}/access_tokens` answers 422, "The level of access for permissions
requested are not granted to this installation.", for the job's whole permission set (measured on
2026-11-17 against an installation with `actions: read`). So every tick fails at the step that
mints the token, not only the re-run, and no state is saved; the watchdog reports each base with
work after 90 minutes. A re-run GitHub refuses for any other reason ends the `checks-red` detail
with `; re-run refused (<status>)`, and the batch is classified in the same tick.
**Known limits.** These are accepted trade-offs of the current design, not bugs to work around:
@@ -1419,10 +1421,11 @@ action.
contains a PR touching a workflow would be rejected. **Actions** write lets the tick re-run the
failed jobs of a red candidate (`flakyReruns`), and the `tick` job of the workflow asks for it
when it mints the App token (`permission-actions: write`). Grant "Actions: Read and write" to
the App, and accept the new permission on the installation, **before** the version of the queue
that re-runs failed jobs reaches the default branch. What happens in the other order has not
been measured yet (a rehearsal will record it): an installation without that permission may
refuse to mint any token, which would fail every tick, not only the re-run. The watchdog does
the App, and accept the new permission on the installation, **before** a tick runs with this
version of the workflow: before OMNI_QUEUE_ENABLED is set or, where the queue is already
enabled, before this version reaches the default branch. In the other order GitHub refuses to
mint the tick's token (HTTP 422 for the job's whole permission set, measured on 2026-11-17), and
every tick fails at that step, not only the re-run. The watchdog does
not use the App: its workflow's own token has `contents: read`, `issues: write` and
`pull-requests: read`.
@@ -2265,8 +2268,9 @@ entries of each list while either is not empty. Check, in order:
1. Why GitHub refuses the App's status writes: the run's log names the error of each refused
write, and so does an `error` event (`<batchId>: could not withdraw …`, or the detail of the
failed tick's own `error` event when the tick threw). A revoked `statuses: write` permission of
the App, an uninstalled App or a GitHub incident are the usual causes.
failed tick's own `error` event when the tick threw). A GitHub incident is the usual cause. A
permission the App no longer holds, or an App that is no longer installed, does not get this
far: GitHub mints no token for the tick, which fails at that step and stores nothing.
2. Once that is fixed, wait until both lists are gone from `state/<sanitized-base>.json` on
`omni-queue-state` (the next ticks write the `failure` statuses). Nothing else needs a person.
3. Then `npm run queue -- resume <base>`. It is refused while `landWithdrawalsOverflow` holds