Skip to content

Scaffolded templates put @ref LLP comments above the frontmatter, silently killing every key (alwaysRun, model, description) #60

Description

@brentvatne

Summary

Every markdown template we ship starts with an @ref LLP … HTML comment placed above the opening ---. The frontmatter parser requires --- at byte 0, so in a freshly scaffolded repo the frontmatter of the coordinator and all three agents parses as nothing at all. Every key is silently dropped — no error, no warning, no doctor complaint.

Found while scaffolding expo/expo with 0.10.0.

Root cause

src/config/load.ts:386:

export function parseFrontmatter(md: string): { data: Record<string, string>; body: string } {
  if (!md.startsWith("---")) {
    return { data: {}, body: md };
  }

Any leading byte — including a comment we ourselves put there — returns { data: {} } and hands the whole file back as body.

All four shipped templates trip it:

$ for f in templates/coordinator.md templates/agents/*.md; do printf "%-40s %s\n" "$f" "$(head -c 3 "$f")"; done
templates/coordinator.md                 <!-
templates/agents/consistency.md          <!-
templates/agents/correctness.md          <!-
templates/agents/security.md             <!-

Impact

Consumed at src/config/load.ts:173-175, so the dropped keys fall back to defaults:

Template Key lost Consequence
agents/security.md alwaysRun: true The security agent stops being always-run. It is now only included when the router happens to pick it.
agents/security.md model: anthropic/claude-opus-5 Security reviews run on the config default instead of Opus.
coordinator.md model: anthropic/claude-opus-5 Consolidation and the final decision run on the default model.
all three agents description The router prompt renders (no description) for every agent — see src/core/prompts.ts:726. Routing has nothing to discriminate on.

alwaysRun is the serious one: src/config/load.ts:174 tests the value against "", so it becomes false, and the guarantee that security always reviews every PR is quietly gone.

Reproduction

Scaffold a fresh repo and run any review. Observed on a real run against expo/expo PR #48510:

  [coordinator] started claude-sonnet-5
Models used — … coordinator: anthropic/claude-sonnet-5

despite model: anthropic/claude-opus-5 sitting in coordinator.md.

Why it went unnoticed

This repo's own .expo-code-review/ files are correct — they start with ---. So do the hand-tuned configs in expo/eas-cli and brentvatne/euxy. Only a freshly scaffolded repo gets the broken shape, and the failure is silent, so nothing surfaces it.

Suggested fix

  1. Move the @ref LLP annotations in templates/coordinator.md and templates/agents/*.md to below the closing ---, or into the frontmatter as # comment lines (the parser already allows those).
  2. Consider making parseFrontmatter tolerate leading whitespace and HTML comments before the delimiter. It is a small change and stops this whole class of silent misconfiguration for adopters who add a license header or a comment of their own.
  3. Add a doctor check: if a prompt file contains a line matching ^(description|alwaysRun|model|temperature): that the parser did not pick up, fail with a pointer to the delimiter. This is the check that would have caught it.

A regression test asserting that every shipped template's frontmatter parses to a non-empty key set would keep it from coming back.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions