openCaselist SyncCopy Anchor Link

Pulls the bulk evidence archives openCaselist publishes — the weekly ZIPs of Verbatim .docx files behind https://opencaselist.com/{caselist}/downloads — and unpacks them into card HTML the rest of the app can index.

• Route: none. This is a sync script, not a page.
• Package: debate-data-sync (src/caselist/), with the DOCX conversion borrowed from debate-card-parser.
• Command: bun run sync-caselist (add --ingest to unpack).

What it coversCopy Anchor Link

Every caselist openCaselist runs, all of which serve the same page and the same bucket layout:

Each publishes one whole-season dump ({slug}-all-{date}.zip) and one delta per week ({slug}-weekly-{date}.zip), cut at midnight every Tuesday morning and served from a public Backblaze bucket that needs no credentials.

How discovery worksCopy Anchor Link

The downloads page is a client-rendered React app. A plain GET returns the <div id="root"> shell and a script tag — the archive links only exist after the bundle runs — so a scraper that reads the raw HTML finds nothing and cannot distinguish that from a caselist with no archives yet. Discovery therefore tries three sources in order and records which one answered:

1. api — the JSON endpoint the page's own client calls.
2. html — the rendered page, when served pre-rendered or when the caller supplies markup it rendered itself.
3. probe — the archive URL is fully determined by the slug and the date, so the candidates for the last N Tuesdays are generated and HEAD-checked. Slower, but independent of openCaselist's markup and API.

A run that finds nothing returns source: "none" plus the reason each source gave, rather than throwing: one caselist being unreachable must not cost a five-caselist run the other four.

How ingest worksCopy Anchor Link

The manifest at data/metadata/caselist-downloads.json doubles as sync state — each caselist lists the archive URLs already ingested, so a run only fetches the weeks added since the last one. A caselist with nothing synced is seeded from the season dump; after that only the weekly deltas are pulled, since they carry the same files for a fraction of the bytes. An archive is marked synced only after its ingest succeeds, so a failure retries exactly that archive.

Unpacking walks the ZIP one entry at a time and hands each converted document straight to its consumer, so a several-hundred-megabyte dump is never held in memory whole. Each document carries the school, team and side read out of its path inside the archive. One unreadable file — password-protected, truncated, a .doc renamed to .docx — is recorded with a coded reason and the walk continues.

Known gapsCopy Anchor Link

• Nothing is persisted yet. The sync discovers, downloads and converts; wiring the converted documents into the evidence library's D1 tables (as the Parquet importer does) is not built. --ingest today proves the pipeline and reports counts.
• No schedule. There is no cron calling the sync — it is run by hand. Archives are cut weekly, so a Wednesday cron would be the natural cadence.
• Path-derived provenance is heuristic. School, team and side come from the layout of paths inside the archive and degrade to null when the layout is unfamiliar, rather than guessing.
• The API endpoint is unverified against production. The JSON source is attempted first and falls through cleanly when it does not answer, so the sync works either way — but the endpoint it tries has not been confirmed against a live openCaselist deploy.