testing-llm
Installation
SKILL.md
LLM & AI Testing Patterns
Patterns and tools for testing LLM integrations, evaluating AI output quality, mocking responses for deterministic CI, and applying agentic test workflows (planner, generator, healer). Of that trio only the healer keeps a local reference here; the planner and generator stages belong to the testing-e2e skill.
Quick Reference
| Area | File | Purpose |
|---|---|---|
| Rules | rules/llm-evaluation.md |
DeepEval quality metrics, Pydantic schema validation, timeout testing |
| Rules | rules/llm-mocking.md |
Mock LLM responses, VCR.py recording, custom request matchers |
| Reference | references/ork-delta.md |
House rules the vendor docs do not carry: GEval and RAGAS API corrections, threshold direction, cassette path, golden-dataset and latency budgets |
| Reference | references/healer-agent.md |
Auto-fixes failing tests (selectors, waits, dynamic content) |
| Reference | references/langfuse-v4.md |
Langfuse Python SDK v4 tracing and dataset runs, plus the non-zero throughput assertion |
| Checklist | checklists/llm-test-checklist.md |
Complete LLM testing checklist (setup, coverage, CI/CD) |