Compaction script (P0 restart-safe migration):
- Derives stable run ID from hash of sorted source file paths
- Archives stored under versioned archive/legacy/<runId>/ path
- Manifest includes full sourceFilePaths list for resume detection
- On rerun: detects existing manifest, verifies archives, resumes
deletion from the original source list — never rebuilds from
a partial file set
- Verifies ALL artifacts (manifest + archives + invalid-records)
- Post-deletion relist confirms every manifest source is gone
- Idempotent upload: skips if manifest checksum already matches
Telemetry writer (memBufferFirstTs bug):
- On flush failure, restore snapshotTs instead of Date.now() so
cross-midnight retries don't change partition paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Telemetry writer: derive partition dates from batch timestamps (KV key
timestamp / memBufferFirstTs) instead of `new Date()` so retries always
produce the same path.
Compaction script: full rewrite addressing all audit P0/P1 findings:
- Paginated listing (follows rel="next" until all 10k+ files listed)
- Fail-closed: aborts entirely if ANY download fails
- Remote verification: re-downloads archives and checks SHA-256 before deleting sources
- Batched deletes: max 50 operations per HF commit
- Preserves FULL invalid lines in separate artifact (not truncated)
- Dry-run is default; --apply required for writes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The HF dataset's telemetry/ folder hit the 10k file limit.
Code fix: writes now target incoming_v2/YYYY/MM/DD/ instead of
telemetry/, distributing files across date-partitioned subdirectories.
Max ~200 files per day-folder means this can never saturate again.
Compaction script: scripts/compact-hf-telemetry.mjs reads the
existing 10k legacy files, groups events by UTC month, and writes
~12 archive files to archive/legacy/. Dry-run by default; --apply
commits archives and deletes legacy files. Verifies event counts
before any deletion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace request.text() with a streaming reader that cancels at
512 KiB — prevents a headerless or chunked-encoding request from
buffering an unbounded payload into Worker memory. No WAF rule
dependency.
Also updates the file header and wrangler.toml comments to
describe the current in-memory fallback (not the old direct-commit
behavior).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When TELEMETRY_KV is not bound, telemetry events now accumulate in
the Worker isolate's module-level buffer (same persistence model as
the rate limiter) and flush to HF when the buffer reaches 50 events,
256 KiB, or 15 minutes — whichever comes first. This eliminates the
one-commit-per-request explosion without requiring any Cloudflare
Dashboard configuration.
KV is still preferred (durable across isolate evictions), but the
fallback now produces ~1 HF commit per 50 requests instead of 50.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P1 fixes:
- Per-request body capped at 512 KiB (was 24 MiB — would OOM Workers)
- Flush reads at most 50 keys capped at 5 MiB combined (was unbounded)
- Dual rate limiting: per-session + per-IP (blocks session_id rotation)
- Content-Length early reject before reading body into memory
- Chunked UTF-8 base64 encoding (avoids Array.from() on full payload)
- Max 100 events per batch (was 500)
- Rate limiter map eviction at 10k entries (prevents isolate memory leak)
- Direct-commit fallback warns loudly when TELEMETRY_KV not bound
- Deployment checklist in header comment (WAF, IP rate limit, KV plan)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live app (index.html):
- Venice API key field in Settings → API Keys tab
- 44 Venice models (VENICE_MODELS + VENICE_TIER_SIZES) race alongside
OpenRouter in ULTRAPLINIAN when a Venice key is set
- queryModel() accepts provider param — routes to Venice or OpenRouter
endpoint with correct auth headers
- veniceApiKey added to state, STATE_KEYS, save/load cycle
Telemetry:
- Reject batches exceeding 24 MiB with 413 before KV.put() (CF KV
values cap at 25 MiB)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Live app (index.html):
- Remove 11 stale OpenRouter models (grok-code-fast-1, both MiMo,
claude-3.5-sonnet, gpt-5.4-chat, gemini-3-pro-preview,
llama-3.1-405b, grok-4, both grok-fast variants, qwq-32b)
- Add replacements: grok-4.5, claude-opus-4.8/fable-5/sonnet-5,
gpt-5.3-5.6 series, qwen3.7-max, glm-5.2, gemini-3.5-flash, etc.
- Update TIER_SIZES to {fast:12, standard:27, smart:41, power:53, ultra:60}
- Update both model dropdowns (header + settings)
- Migrate HALL_OF_FAME combo from dead claude-3.5-sonnet to claude-sonnet-4.6
Telemetry:
- Replace btoa() with UTF-8-safe base64 encoding (btoa crashes on
non-Latin-1 characters in telemetry JSONL)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Concurrent flushes reading identical KV keys now produce the exact same
filename (`flush_<sha256>.jsonl`), making HF overwrites truly idempotent.
The previous approach included a timestamp which broke idempotency — two
isolates hitting the threshold simultaneously would write different files
containing duplicate data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Require Bearer TELEMETRY_FLUSH_SECRET on GET ?flush=force
- Derive flush filename from key fingerprint (concurrent flushes → same
file → idempotent overwrite, no duplicate rows)
- Fix TS errors in Venice helper (explicit type assertions)
- Correct model counts in SettingsModal (smart=41, power=53, ultra=60)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
KV lock removed — KV is eventually consistent and can't do atomic CAS.
Concurrent flushes may produce duplicate JSONL files on HF, which is
harmless (extra telemetry rows, not data loss or corruption).
Added GET /api/telemetry?flush=force so an external cron can trigger
guaranteed periodic flushes regardless of traffic volume.
Added .gitignore (repo had none).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Streaming path now returns diagnostic message (not blank) when all models fail
- All models_queried/models_total counts include Venice entries
- KV flush uses a 60s lock to prevent concurrent flush races
- TTL increased from 24h to 7 days (low-volume batches won't expire)
- Restored all previously-existing telemetry ALLOWED_FIELDS (winner_combo,
stream, fast_stream, has_image, liquid_upgrades, encoding, etc.)
- Replaced AbortSignal.any with compatible signal forwarding pattern
- wrangler.toml no longer has fake namespace ID (commented out for local dev)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update ULTRAPLINIAN model registry: remove 12 dead OpenRouter models, add frontier models (grok-4.5, glm-5.2, qwen3.7-max, claude-opus-4.8, gpt-5.5/5.6, etc.)
- Add Venice API as a second provider: race Venice models alongside OpenRouter in ULTRAPLINIAN mode when a Venice key is configured
- Replace generic "All models failed" with specific diagnostics (auth, credits, not_found, rate_limit)
- Buffer telemetry in Cloudflare KV and flush as bulk commits to prevent HF commit explosion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>