listAllFiles('archive/legacy') throws 404 EntryNotFound when the
directory doesn't exist yet on HF. Catch it so the first run
proceeds instead of dying at Step 2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
HF tree API defaults to non-recursive, so nested manifest.json files
under archive/legacy/<runId>/ were invisible. After a partial deletion,
the script would miss the original manifest and incorrectly rebuild
archives from the reduced file set.
Now listAllFiles accepts a recursive flag, and the archive scan uses it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Compaction script (P0 restart-safe migration):
- Derives stable run ID from hash of sorted source file paths
- Archives stored under versioned archive/legacy/<runId>/ path
- Manifest includes full sourceFilePaths list for resume detection
- On rerun: detects existing manifest, verifies archives, resumes
deletion from the original source list — never rebuilds from
a partial file set
- Verifies ALL artifacts (manifest + archives + invalid-records)
- Post-deletion relist confirms every manifest source is gone
- Idempotent upload: skips if manifest checksum already matches
Telemetry writer (memBufferFirstTs bug):
- On flush failure, restore snapshotTs instead of Date.now() so
cross-midnight retries don't change partition paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Telemetry writer: derive partition dates from batch timestamps (KV key
timestamp / memBufferFirstTs) instead of `new Date()` so retries always
produce the same path.
Compaction script: full rewrite addressing all audit P0/P1 findings:
- Paginated listing (follows rel="next" until all 10k+ files listed)
- Fail-closed: aborts entirely if ANY download fails
- Remote verification: re-downloads archives and checks SHA-256 before deleting sources
- Batched deletes: max 50 operations per HF commit
- Preserves FULL invalid lines in separate artifact (not truncated)
- Dry-run is default; --apply required for writes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The HF dataset's telemetry/ folder hit the 10k file limit.
Code fix: writes now target incoming_v2/YYYY/MM/DD/ instead of
telemetry/, distributing files across date-partitioned subdirectories.
Max ~200 files per day-folder means this can never saturate again.
Compaction script: scripts/compact-hf-telemetry.mjs reads the
existing 10k legacy files, groups events by UTC month, and writes
~12 archive files to archive/legacy/. Dry-run by default; --apply
commits archives and deletes legacy files. Verifies event counts
before any deletion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>