QA Agent Testing

Design and run reliable evaluation suites for LLM agents, including tool-using, multi-turn, and multi-agent systems.

For acceptance, review burden, rework, and cost on selected backlog tasks, use the backlog benchmark protocol and fillable template. Otherwise use the workflow below.

Default QA Workflow

1. Define the Agent Under Test (AUT): scope, tools, approval boundaries, out-of-scope requests, and safety rules.
2. Build a starter suite from real work:
   • Smoke suite: 5-8 highest-signal checks for PR gates
   • Regression suite: 15-25 tasks from real failures, tickets, or production traces
   • Refusal/security pack: unsafe requests, prompt injection, tool-output poisoning, and exfiltration attempts
   • Iterative coding trajectory: evolving specifications applied to the agent's own carried workspace, when extension quality matters
3. Define objective graders first: schema checks, golden traces, deterministic mocks, policy oracles, and tool side-effect checks.
4. Add model-based graders only where objective checks are insufficient. A judge earns a place on a gate only after the calibration gate in llm-judge-limitations.md; until then its verdicts are triage, not status. Never let a judge grade refusals or policy hard fails: those are oracle checks.
5. Run offline evals with deterministic controls and trace logging.
6. Add optional online evals or canary comparisons for live traffic.
7. Gate changes on one consistent status model and log regressions.