Fantasy Flu

FLU's one unbreakable rule is "show the data, don't editorialize," and an LLM will break it by default. So the guardrails are part of the system design, not left to the prompt alone: • Grounding: the model only sees a structured fact sheet with stable IDs, and slides that don't cite it fail validation. • Retry, then fail safe: a malformed or failed story gets one resubmit. After that it's marked failed, and the weekly digest goes out without a story. • Human override: admins can pull a published story and restore it later. The original is kept for audit. • Cost and access gates: generation only runs for leagues entitled to it, and the pipeline does nothing without an API key.

Validation catches broken formatting and engagement data catches boring stories, but neither catches a story that reads well and quietly breaks a house rule. That needed a judge. A second model grades every new story against its own fact sheet on a seven-check rubric: every number cited, no verdicts, no invented chronology, the right team credited for each figure, and house style. It returns a 0-100 score and quotes the exact text behind each failed check, so failures are traceable instead of hallucinated. The judge is versioned like an instrument. Each rubric change gets a new version so scores stay comparable, and the rubric changed based on real runs. One version learned that vivid "broadcast color" is the point, not a verdict. Another added a check after a story cited a correct number but credited it to the wrong team. When the smaller model kept flagging allowed writing, I moved the judge to a more capable model with extended thinking: the judge benefits from more reasoning, while the writer is matched to the cost of running every week.

I run FLU like a product org where the engineers are AI agents. My job is the product leader's job: decide what to build and why, write it down clearly, and review the result. • Written context: a living CLAUDE.md documents the product rules, architecture, and API quirks, so every agent session starts with the same understanding. • Plan, then build: each feature starts as a plan doc (problem, rulings, PR sequence) and a design brief, then ships as a series of small, reviewable PRs. • Parallel agents: independent work runs in parallel sessions on separate git worktrees. • Quality gates: CI runs typecheck, lint, tests, and a bundle-size budget on every PR, so agent output is held to the same bar as human code. The stack: Python and FastAPI, SQLAlchemy and Postgres, a durable background worker, React with TanStack Router and Query, Tailwind, Stripe, Web Push, and passkeys, deployed on Railway.