RAG Evaluation Harness

Use this skill to measure a retrieval-and-citation contract without making model calls or network requests. The bundled evaluator compares explicit document IDs, so it is suitable for repeatable local checks and CI gates.

Input Contract

Provide one JSON object per line with a unique string id and three arrays of document IDs:

{"id":"question-1","relevant_document_ids":["doc-a"],"retrieved_document_ids":["doc-b","doc-a"],"cited_document_ids":["doc-a"]}

Blank lines are ignored. Invalid JSON, missing arrays, non-string IDs, and duplicate case IDs fail with the JSONL line number. Keep the fixture's relevance labels and citation IDs explicit; do not infer them from answer text.

Run an Evaluation

Set the installed skill directory and run the standard-library-only evaluator: