5 Commits

Author SHA1 Message Date
faber
2e1cf58343 fix: handle missing archive/legacy dir on first compaction run
listAllFiles('archive/legacy') throws 404 EntryNotFound when the
directory doesn't exist yet on HF. Catch it so the first run
proceeds instead of dying at Step 2.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 10:16:56 -04:00
faber
9f714a09fc fix: use recursive listing when scanning archive/legacy for manifests
HF tree API defaults to non-recursive, so nested manifest.json files
under archive/legacy/<runId>/ were invisible. After a partial deletion,
the script would miss the original manifest and incorrectly rebuild
archives from the reduced file set.

Now listAllFiles accepts a recursive flag, and the archive scan uses it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 10:16:56 -04:00
faber
9f4f898b8e fix: resumable compaction + restore memBufferFirstTs on flush failure
Compaction script (P0 restart-safe migration):
- Derives stable run ID from hash of sorted source file paths
- Archives stored under versioned archive/legacy/<runId>/ path
- Manifest includes full sourceFilePaths list for resume detection
- On rerun: detects existing manifest, verifies archives, resumes
  deletion from the original source list — never rebuilds from
  a partial file set
- Verifies ALL artifacts (manifest + archives + invalid-records)
- Post-deletion relist confirms every manifest source is gone
- Idempotent upload: skips if manifest checksum already matches

Telemetry writer (memBufferFirstTs bug):
- On flush failure, restore snapshotTs instead of Date.now() so
  cross-midnight retries don't change partition paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 10:16:56 -04:00
faber
2896b530ac fix: deterministic date partitions + compaction script rewrite
Telemetry writer: derive partition dates from batch timestamps (KV key
timestamp / memBufferFirstTs) instead of `new Date()` so retries always
produce the same path.

Compaction script: full rewrite addressing all audit P0/P1 findings:
- Paginated listing (follows rel="next" until all 10k+ files listed)
- Fail-closed: aborts entirely if ANY download fails
- Remote verification: re-downloads archives and checks SHA-256 before deleting sources
- Batched deletes: max 50 operations per HF commit
- Preserves FULL invalid lines in separate artifact (not truncated)
- Dry-run is default; --apply required for writes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 10:16:56 -04:00
faber
70cb3bb188 fix: date-partition telemetry writes + legacy compaction script
The HF dataset's telemetry/ folder hit the 10k file limit.

Code fix: writes now target incoming_v2/YYYY/MM/DD/ instead of
telemetry/, distributing files across date-partitioned subdirectories.
Max ~200 files per day-folder means this can never saturate again.

Compaction script: scripts/compact-hf-telemetry.mjs reads the
existing 10k legacy files, groups events by UTC month, and writes
~12 archive files to archive/legacy/. Dry-run by default; --apply
commits archives and deletes legacy files. Verifies event counts
before any deletion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 10:16:56 -04:00