---
title: "ADR-002: content operations"
description: "Automated content supply, licensed sources and AI originals, QA, auto-publish, the admin API."
source: "docs/decisions/ADR-002-content-operations.md"
---

> Part of [Decisions](/decisions/llms.txt). Every page: [llms.txt](/llms.txt).

# ADR-002 — Content operations: automated supply, licensed sources + AI originals, admin dashboard

Status: Accepted 2026-10-17. Owner: TPM. Complements ADR-001 (D7 batches, D9 difficulty tiers, D13 languages).

## Goal

Keep every (language × topic × difficulty) cell stocked with fresh, legally safe, retell-worthy articles **without a human writing or approving each one**. Humans only (a) tune sources, prompts and thresholds, (b) clear a small review queue, (c) react to user reports. Everything else runs on schedule.

## D1. Two supply lines, one article shape

| Line | How | Rights |
|---|---|---|
| **Licensed adaptation** | Fetch from registered sources (RSS/API/HTML), extract the text, have the LLM adapt it into a short argumentative piece in the target language (the article is a rewrite that keeps the claims, not a copy), generate key points, reading hint, topic, difficulty. | Only sources whose license is on the allowlist: public domain / CC0, CC BY, CC BY-SA (adaptation published under the same license with attribution), government works in the public domain, and sources the owner has explicitly cleared. Attribution text + URL + license are stored and shown in the app's article footer. |
| **AI original** | Generate from a topic brief (topic × angle × difficulty × language) with a prompt library; no external source. | Owned content. Labelled `source.kind = "ai_original"` and byline "编辑部 / Editorial" + "AI 辅助创作 / AI-assisted". |

Both produce the same `Article` (ADR-001 contract) plus `source {kind, license, attribution, url, retrieved_at, source_id}` and `provenance {generator_model_ref, qa_model_ref, prompt_version, job_id}`.
Bilingual **pairs** are generated when a piece works in both languages (`pair_id`), so the same claims exist in en and zh-Hans; pairs are independent articles for the app.

Initial source allowlist (operator can add more in the dashboard; each needs `license` + `attribution_template`): Wikinews, Wikipedia (CC BY-SA 4.0), Wikivoyage, PLOS journals (CC BY), Project Gutenberg / Standard Ebooks (PD), US federal agency sites such as NASA/NIH (PD), Our World in Data (CC BY), 中文维基百科/维基新闻 (CC BY-SA), CC-licensed newsletters and blogs the owner registers. Sources that are "all rights reserved" cannot be registered (the dashboard refuses a license outside the allowlist).

## D2. Pipeline (Cloudflare cron + Queues)

1. **Inventory check** (hourly): for each cell (language × topic × difficulty) compute `unseen_available` = published articles not completed by most active users (proxy: published count minus median completions). If below `target` (default 20) enqueue generation jobs up to `batch_max` (default 5 per cell per run).
2. **Fetch** (per source schedule, default every 6 h): pull new items, dedupe by URL hash, extract main text (Readability-style), store as `source_items` with `license` inherited from the source. Skip items < 150 words / 250 chars or > 4000 words.
3. **Generate** (queue consumer, uses the dynamic LLM provider chain from ADR-001 D11): adaptation or original → draft article in the target language with `paragraphs`, `title`, `reading_hint`, `key_points` (3–5, weighted), `topic_id`, `difficulty`, `byline`, `source`, `pair` optional.
4. **QA** (second LLM pass + deterministic checks) produces a `quality_report`:
   - Deterministic: length within the tier range for the language (ADR-001 D9 / D13), 3–5 key points, language detected = target, no URLs/emails/phone numbers, no names of private individuals (NER heuristic), duplicate check against published articles via embeddings (Vectorize; cosine > 0.92 = duplicate), attribution present when `kind = licensed`.
   - LLM rubric (0–100 each): `argument_clarity` (one main claim + reasons), `retell_worthiness` (has structure a learner can reproduce), `grounding` (adaptation keeps the source's claims; for originals: internally consistent, no fabricated statistics — numbers must be absent or sourced), `safety` (no hate, sexual content, self-harm, medical/financial advice, defamation; sensitive politics flagged not blocked), `readability` for the tier.
   - `qa_score = min(argument_clarity, retell_worthiness, grounding, readability)`; `safety` is a hard gate.
5. **Decision**: `qa_score ≥ auto_publish_threshold` (default 80) and safety pass and no flags → `published`. `60 ≤ qa_score < 80` or any soft flag → `needs_review` (dashboard queue). `< 60` or safety fail or duplicate → `rejected` (kept 30 days for prompt tuning). Thresholds are per language and editable.
6. **Post-publish signals**: helpfulness votes on assessments, completion rate, average `key_info` score (too low across users ⇒ key points mis-specified), user reports. A nightly job computes `article_health`; `health < 40` with ≥ 20 sessions ⇒ `retired` automatically and a replacement job is enqueued. Reports with reason `rights` or `harmful` unpublish immediately and open a review item.
7. **Spot check**: 5 % of auto-published articles are sampled into the review queue as `audit` items so humans calibrate the threshold without blocking supply.

**Spend guard**: `daily_cost_cap_usd` (default 20) and `max_jobs_per_run` (default 25) bound the pipeline; reaching the cap auto-pauses generation with `paused_reason = cost_cap` (shown on the overview) and resumes the next UTC day. Three consecutive provider failures on `content_llm` pause with `provider_errors`.

Every stage is idempotent, records `model_ref`, prompt version and cost (tokens × provider price table) so the dashboard can show cost per published article.

## D3. Article lifecycle

`draft → qa → needs_review | published | rejected`, `published → retired`, `needs_review → published | rejected` (human), `published → unpublished` (report) `→ published | retired`. Published articles are versioned: an edit creates `version + 1`; sessions pin `article_id + version`; key_points changes re-run QA.

## D4. Dashboard (`dashboard/`)

Web app for operators, separate from the landing page. TanStack Start + React on Cloudflare Workers, talks only to the backend admin API (`docs/api/openapi-admin.yaml`). Auth: Cloudflare Access in production (verify the `Cf-Access-Jwt-Assertion` header against the team's public keys), `X-Admin-Key` in dev. Screens:

- **Overview**: inventory heatmap (language × topic × difficulty vs target), pipeline throughput (24 h / 7 d), auto-publish rate, review-queue size, cost per article, provider error rate, top reports.
- **Review queue**: items needing review or audit; side-by-side source text vs draft; QA report with per-check results; actions: publish, edit-and-publish, regenerate (with a note that feeds the prompt), reject (with reason taxonomy that trains thresholds).
- **Articles**: searchable table with status, language, topic, difficulty, health, sessions, key_info average; detail page = in-app preview (uses design tokens), key points editor, version history, retire/unpublish.
- **Sources**: register/edit sources (URL, type rss|api|html|sitemap, license from allowlist, attribution template, schedule, topic hints, language), fetch now, last run log, yield stats, pause.
- **Generation**: topic briefs library (per language), targets per cell, batch size, auto-publish thresholds, run-now, job log with cost.
- **AI providers**: the ADR-001 D11 chains (ASR/LLM/TTS) plus `content_llm` and `qa_llm` chains, per-provider usage/cost/error charts, a "test prompt" box.
- **Quality**: article health distribution, assessment helpfulness by article, flags, sampled audits and their outcomes, threshold tuning history.
- **Users & reports**: report inbox, user lookup by id (no PII beyond display name/locale), entitlement state, delete-data trigger. Waitlist export.
- **Playground** (amended 2026-10-19, issue #33): operators run the app's 3-step loop (ADR-001 D1) on a published article to compare scoring models. One recording (recorded in the browser or uploaded) and one transcript from one ASR model are scored by 2–6 pinned `llm` lanes at once, side by side: overall, band, the four dimensions with evidence, key-point coverage, the one action, confidence, errors. Round 2 is scored per lane against that lane's round 1; each lane can produce its model answer. Blind by default: lanes are labelled A, B, C… in a random order with opaque lane ids; model names, latency, tokens, cost and finish times stay hidden, and a round's results are released together, until the operator gives a verdict (winner or tie, with a note) or reveals. The first verdict given blind is kept unchanged next to the latest one, and once a run has a verdict its recorded rounds can't be replaced (a new attempt is a new run). Lanes call one model each, with no chain fallback, and use the same validity check (ADR-001 D2) and assessment post-processing (ADR-001 D3) as the app. Runs are stored apart from user data: they never touch users, sessions, attempts, assessments, XP, ability or app statistics. Their provider calls are recorded and priced under the run id and left out of every aggregate (overview cost and provider error rates, provider usage, `daily_cost_cap_usd`); playground audio is purged after 30 days. Comparing ASR models and the `content_llm` / `qa_llm` chains is out of scope for now.

Principle: every list has loading/empty/error states; every destructive action confirms; every automated decision shows *why* (the QA report) so a human can disagree in one click.

## D5. Human-in-the-loop budget

Target: ≤ 15 minutes/day of operator time at steady state. Metrics to prove it: review-queue inflow per day, median time-in-queue, auto-publish share (target ≥ 85 %), retire rate. If inflow exceeds 20/day the dashboard surfaces the top rejection reasons so the prompt/threshold can be tuned instead of reviewing more.