Eval Relevance

Use this skill to evaluate how relevant an assistant response is to the user’s request.

Inputs

Require:

• The assistant response text to evaluate.
• (Optional) The user’s original request for comparison.

Internal Rubric (1–5)

5 = Directly addresses the user’s request, stays fully on-topic, and prioritizes what the user actually asked
4 = Mostly relevant, minor digressions or small omissions
3 = Partially relevant, addresses the general topic but misses key parts of the request
2 = Weak relevance, significant digressions or failure to address the core request
1 = Not relevant, does not address the user’s request or answers a different question entirely

Workflow

1. Compare the assistant response to the user’s request (if provided).
2. Score relevance on a 1-5 integer scale using the rubric only.
3. Write concise rationale tied directly to rubric criteria.
4. Produce actionable suggestions that improve relevance.

Output Contract

Return JSON only. Do not include markdown, backticks, prose, or extra keys.

Use exactly this schema:

{ "dimension": "relevance", "score": 1, "rationale": "...", "improvement_suggestions": [ "..." ] }

Hard Rules

• dimension must always equal "relevance".
• score must be an integer from 1 to 5.
• rationale must be concise (max 3 sentences).
• Do not include step-by-step reasoning.
• improvement_suggestions must be a non-empty array of concrete edits.
• Never output text outside the JSON object.