clerk-evals

Evaluation suites for testing how language models and coding agents write Clerk code. The suite covers 37 tasks across Next.js, React, iOS, and Android. Vercel AI Gateway provides one transport for models from OpenAI, Anthropic, Google, xAI, Moonshot AI, Tencent, DeepSeek, and Z.AI.

Install Bun >=1.3.0, then add a Vercel AI Gateway key. See .env.example.

cp .env.example .env
bun i
bun start

For detailed, copy-pastable steps see docs/ADDING_EVALS.md. In short:

• Create src/evals/your-eval/ with PROMPT.md and graders.ts.
• Implement graders that return booleans using defineGraders(...) and shared judges in @/src/graders/catalog.
• Append an entry to the evaluations array in src/config/evaluations.ts with framework, category, and path.
• Run bun start --eval "your-eval" --smoke --debug to test with one model.

[
  {
    "model": "claude-sonnet-4-5",
    "framework": "Next.js",
    "category": "Auth",
    "value": 0.8333333333333334,
    "updatedAt": "2026-01-06T17:51:27.901Z"
  },
  {
    "model": "gpt-5-chat-latest",
    "framework": "Next.js",
    "category": "Auth",
    "value": 0.6666666666666666,
    "updatedAt": "2026-01-06T17:51:30.871Z"
  },
  {
    "model": "claude-opus-4-5",
    "framework": "Next.js",
    "category": "Billing",
    "value": 1.0,
    "updatedAt": "2026-01-06T17:51:56.370Z"
  }
]

Debugging

# Run a single evaluation with debug output
bun start --eval "auth/routes" --debug

# Smoke test (one model, one eval)
bun start --eval "auth/routes" --smoke --debug

bun start [options]

# Baseline (no tools)
bun start --model "grok-4.6" --eval "protect"

# With MCP tools
bun start --mcp --model "claude-sonnet-5-5" --eval "protect"

# With skills
bun start --skills --model "claude-sonnet-5-5"

# Local MCP server
MCP_SERVER_URL_OVERRIDE=http://localhost:8787/mcp bun start --mcp

# Dry run (see what would execute)
bun start --dry

bun start --models "deepseek-v4.1-flash,glm-5.3-flash,claude-opus-5-5,claude-sonnet-5-5,gpt-6-astra,gpt-6-sol,gpt-6-luna,gpt-6.1-sol,grok-4.7"

Each task has a five-minute limit by default. Use --timeout for one run or set EVAL_TASK_TIMEOUT_MS to change the default. Direct eval runs allow 32,768 output tokens by default. Use --max-output-tokens or EVAL_MAX_OUTPUT_TOKENS to change that limit. Add --debug to print task timing, step progress, and redacted gateway error details. Use --max-retries 0 to isolate one gateway attempt.

Responses that reach the output token limit are graded with their available text. This makes truncation a model-quality result instead of missing data. Transport and gateway failures remain retryable errors.

Retry only the missing cells from a stored run, then rebuild its score file:

bun retry:run "mcp-2026-09-03T19-08-33-670Z" --concurrency 1
bun retry:run "skills-2026-09-03T20-20-09-441Z" --model grok-4.6 --eval ios/custom-setup
bun rebuild:scores "skills-2026-09-03T20-20-09-441Z"

Routine runs include models released in the last 90 days. A creator's model marked as its current best model remains in routine runs after 90 days. OpenAI's current best is GPT-6 Astra, Anthropic's is Claude Opus 5.5, and xAI's is Grok 4.7. Exact --model and --models selections can use old catalog models. Use --include-legacy to run the complete catalog.

The external model set also tracks strong results from the Convex LLM leaderboard. Earlier additions were Grok 4.6, Grok 4.5, Tencent Hy4 Preview, and Kimi K3. These models ranked 4, 8, 9, and 13 when selected on September 3, 2026.

To keep all selected models under one run ID in each mode, use bun start --models, bun start:mcp --models, and bun start:skills --models. Use bun retry:run with that run ID to retry missing tasks.

Run all configured models sequentially with timeout and retry:

./run-evals.sh                              # Default models, baseline + MCP
./run-evals.sh --models "gpt-5,claude-sonnet-5-5"  # Specific models
./run-evals.sh --include-legacy             # Complete catalog
./run-evals.sh --baseline-only              # Skip MCP
./run-evals.sh --mcp-only                   # Skip baseline
./run-evals.sh --list                       # List available models

Set BRAINTRUST_API_KEY to enable experiment logging and tracing:

# Single run: creates experiment automatically
BRAINTRUST_API_KEY=sk-... bun start --mcp

# Batch run: defers reporting, consolidates into one experiment per mode
BRAINTRUST_API_KEY=sk-... ./run-evals.sh

# Manual consolidated report from recent results
BRAINTRUST_API_KEY=sk-... bun report:braintrust --since "2026-03-19T17:00:00Z"

The eval runner uses wrapAISDK to auto-trace all generateText calls (inputs, outputs, tool invocations, token usage). Traces flow to Braintrust even during batch runs.

Run evaluations using AI coding agents (Claude Code, Codex) instead of direct LLM calls.

Agent evals spawn CLI tools as child processes and grade the final isolated workspace. Install a supported CLI before running:

• Claude Code
• Codex CLI

Set VERCEL_AI_GATEWAY_API_KEY in .env. The harness maps this key to each CLI. Direct provider keys remain a fallback for local compatibility. Pin the model with --model or ANTHROPIC_MODEL for Claude Code and OPENAI_MODEL for Codex.

Registered agent tasks use an explicit repository fixture. A task can also define hidden Bun tests. The harness stages those tests outside the repository after the coding agent exits. Test failure is a hard score gate, while normal deterministic graders still provide diagnostic partial credit.

The add-auth eval is the exception to the repository's checked-in PROMPT.md convention. It fetches skills/clerk-setup/SKILL.md from an immutable clerk/skills revision at runtime so the baseline and Skills columns always evaluate the canonical setup prompt. Runs resolve clerk/skills main once at startup and record that commit, or evaluate a specific commit:

CLERK_SKILLS_SHA=<full-clerk-skills-sha> bun agent:claude --eval add-auth

With --skills, the Skills checkout must be at the evaluated commit.

Eval runs are single-turn, so the harness puts a short context before the skill body: the user approved the setup checklist but declined installing Clerk's agent skills, and the agent shouldn't wait for replies or leave a dev server running. The skill body itself is unchanged.

.github/workflows/add-auth-automation.yml accepts clerk_skills_updated dispatches with client_payload.sha, clerk_cli_released dispatches with client_payload.version, and manual runs. Manual runs can select a full Skills SHA and the current latest CLI version. The canonical prompt runs npx -y clerk@latest init, so a requested CLI version that isn't clerk@latest fails instead of mislabeling the result. A CLI release waits up to 10 minutes for its version to become clerk@latest; if a newer stable release has already superseded it, the older run skips quietly.

The workflow runs the four add-auth variants twice with the same prompt and Clerk CLI version. Baseline receives no installed Clerk skills. The Skills column checks out the exact CLERK_SKILLS_SHA, verifies the checkout, and makes its clerk-setup skill available to the agent. Both columns upload scores and source-version metadata. A missing score or setup failure fails the automation and alerts #team-docs through DOCS_SLACK_WEBHOOK_URL; a completed Skills score below baseline is instead reported as an eval regression warning.

Artifacts are named by Skills SHA and CLI version, and baseline artifacts also carry the Claude Code version. Each run installs the latest Claude Code version, pinned for both columns. Before evaluating, the workflow diffs the incoming Skills SHA against the last evaluated one:

• If no skill the add-auth Skills run loads changed (the evals/add-auth entry in src/config/skills.ts), the run skips. Edits to unrelated skills don't change the score.
• If skills/clerk-setup/ is unchanged and a stored baseline used the same Clerk CLI, Claude Code version, and harness commit, the run reuses that baseline and evaluates only the Skills column. The comparison records which run and Skills SHA each column came from.
• Otherwise, or when the history is missing or unclear, both columns run.

A revision that already has Skills scores and a comparison skips; a manual run with force always runs both columns. There is no mutable “latest result” file, so an older run cannot overwrite a newer result. Artifacts expire after 90 days, after which the next run starts fresh. The repository needs VERCEL_AI_GATEWAY_API_KEY and DOCS_SLACK_WEBHOOK_URL Actions secrets before the first live run. This workflow depends on skills/clerk-setup/SKILL.md existing at the evaluated revision (the flat path from clerk/skills#83).

bun start:agent --agent claude-code --model claude-sonnet-5-5 [options]

Shortcuts:

bun agent:claude --model claude-sonnet-5-5
bun agent:claude:mcp --model claude-sonnet-5-5
bun agent:codex --model gpt-5.6-sol

Examples:

# Run all evals with Claude Code
bun start:agent --agent claude-code --model claude-sonnet-5-5

# Run specific eval with debug output
bun start:agent -a claude-code -m claude-sonnet-5-5 -e add-auth -d

# Run with MCP tools enabled
bun start:agent --agent codex --model gpt-5.6-sol --mcp

bun start              # 1. Baseline -> scores.json
bun start --mcp        # 2. MCP -> scores-mcp.json
bun merge-scores       # 3. Merge -> llm-scores.json

The merge script combines both score files and calculates improvement metrics:

{
  "model": "claude-sonnet-4-5",
  "label": "Claude Sonnet 4.5",
  "framework": "Next.js",
  "category": "Auth",
  "value": 0.83,
  "provider": "anthropic",
  "mcpScore": 0.95,
  "improvement": 0.12
}

This project is broken up into a few core pieces:

• src/index.ts: This is the main entrypoint of the project. Models, reporters, and the runner are registered here, and all executed. Evaluations are defined in src/config/evaluations.ts.
• /evals: Folders that contain grading expectations and, except for add-auth, a checked-in prompt. The add-auth prompt is fetched from the canonical clerk/skills revision set by CLERK_SKILLS_SHA, or clerk/skills main when it's unset.
• /runners: The primary logic responsible for loading evaluations, calling provider llms, and outputting scores.
• /reporters: The primary logic responsible for sending scores somewhere — stdout, a file, etc.

A runner takes a simple object as an argument:

{
  "provider": "openai",
  "model": "gpt-5",
  "evalPath": "/absolute/path/to/clerk-evals/src/evals/auth/protect",
}

It will resolve the provider and model to the respective SDK.

It will load the designated evaluation, generate LLM text from the prompt, and pass the result to graders.

At the moment, evaluations are simply folders that contain:

• PROMPT.md: the instruction for which we're evaluating the model's output on. The add-auth eval instead fetches the canonical clerk-setup skill at runtime.
• graders.ts: a module containing grader functions which return true/false signalling if the model's output passed or failed. This is essentially our acceptance criteria.

Shared grader primitives live in src/graders/index.ts. Use them to declare new checks with a consistent, terse shape:

import { contains, defineGraders, judge } from '@/src/graders'
import { llmChecks } from '@/src/graders/catalog'

export const graders = defineGraders({
  references_middleware: contains('middleware.ts'),
  package_json: llmChecks.packageJsonClerkVersion,
  custom_flow_description: judge(
    'Does the answer walk through protecting a Next.js API route with Clerk auth() and explain the response states?',
  ),
})

• contains / containsAny: case-insensitive substring checks by default
• matches: regex checks
• judge: thin wrappers around the LLM-as-judge scorer. Shared prompts live in src/graders/catalog.ts; add new reusable prompts there.
• defineGraders: preserves type inference for the exported graders record.

For a given model, and evaluation, we'll retrieve a score from 0..1, which is the percentage of grader functions that passed.

Three reporters:

• console: color-coded ASCII table (category x model matrix)
• file: saves scores to scores.json / scores-mcp.json / scores-skills.json
• braintrust: logs experiments to Braintrust (opt-in via BRAINTRUST_API_KEY)

For batch runs, src/report-braintrust.ts consolidates all per-model results from SQLite into a single experiment per mode.

For the notable interfaces, see /interfaces.