LLM Challenge

Benchmark for @tailor-platform/sdk AI-friendliness. Located at llm-challenge/.

Read llm-challenge/README.md for commands, scoring, and verification details.

Core Rule

When AI fails a challenge, improve the SDK (JSDoc, error messages, types, CLAUDE.md) — NEVER add hints to problem.md.

Prerequisite

ALWAYS build SDK before running: pnpm -C packages/sdk build

Problem Conventions

Structure: problems/<id>-<name>/ with meta.json, problem.md, scaffold/, solution/, tests/

meta.json rules:

• id: 3-digit zero-padded, sequential
• scoring: Category defaults — tailordb: 20/20/60, resolver/executor/workflow: 15/15/70, config: 30/20/50, fix-broken: 15/15/70
• Fix-broken problem: same file appears in both implement and scaffold

problem.md rules:

• Sections: Goal → Domain Context/Instructions → What to Build → Requirements → Reference
• NEVER include SDK code examples — AI must discover API from the SDK package itself
• Always end with "Refer to the installed SDK package for ..."

Writing Tests

• Read existing tests in problems/*/tests/ for patterns
• Helpers: shared/test-helpers.ts (createWorkDirContext, importPath, expectFieldType, etc.)
• Mocks: shared/mocks.ts (setupTailordbMock, setupWorkflowMock)
• ALWAYS use describe.skipIf(!workDirReady) guard

Creating a New Problem

1. Next sequential ID (e.g., 013)
2. Write solution first, then tests
3. Verify: pnpm -C llm-challenge challenge --problem <id> --use-solution → must be 100/100

SDK Improvement Cycle

1. pnpm -C llm-challenge challenge:solve --retry 2 → analyze failures
2. Improve SDK source (NOT problem descriptions)
3. pnpm -C packages/sdk build → pnpm -C llm-challenge challenge:verify-solution
4. Re-run benchmark to measure improvement