rag-evaluation
rag-evaluation
Covers the non-obvious measurement traps, metric choices, and eval infrastructure decisions for RAG pipelines. Assumes you have a working RAG pipeline and want to measure it.
1. The Three Failure Modes and Which Metrics Catch Them
RAG fails in three distinct ways. Each requires different metrics — conflating them produces misleading aggregate scores.
| Failure mode | Example | Metric |
|---|---|---|
| Retrieval misses relevant doc | Right answer exists, never retrieved | Recall@k |
| Retrieval returns irrelevant docs | Retrieved docs don't support the answer | Precision@k, Context Relevance |
| Generation hallucinates | Docs retrieved correctly, answer fabricated | Faithfulness, Answer Grounding |
Non-obvious: A high faithfulness score with low retrieval recall is a trap — the model is faithfully generating from the wrong context. Always report retrieval and generation metrics together; never report faithfulness alone.
More from blunotech-dev/agents
anti-purple-ui
Enforce a strict monochrome UI with a single high-contrast accent color, removing generic tech gradients and “AI-style” palettes. Use when the user wants minimal, anti-AI, or non-generic aesthetics, or says the UI looks too techy or generic.
9harmonize-whitespace
Align all spacing (padding, margins, gaps) to a consistent 4pt/8pt grid. Use when spacing feels off, inconsistent, cramped, or unbalanced, or when the user asks for a spacing scale or alignment fix.
9typographic-hierarchy
Improve typography by adjusting font sizes, weights, spacing, and contrast to create clear visual hierarchy and readability. Use when text feels flat, unstructured, or when the user asks to refine headings, type scale, or overall readability.
7micro-interaction-adder
Add polished CSS micro-interactions like hover effects, transitions, and feedback states to improve UI feel. Use when the user asks for animations, better UX, or when the interface feels static, plain, or unresponsive.
4consistent-border-radius
Normalizes rounded corners across a file so buttons, inputs, cards, modals, badges, and all UI elements share the exact same curvature. Use this skill whenever the user mentions inconsistent border radii, wants to unify rounded corners, asks to make UI elements look more cohesive, or says things like "make the corners match", "fix the rounding", "unify border radius", "standardize my rounded corners", or "buttons and cards don't match". Also trigger when the user pastes a CSS/HTML/JSX/TSX file and asks for a design consistency pass, border radius is one of the first things to normalize.
4component-split
Analyze a component and determine when and how to split it based on size, responsibility, and reuse signals, producing a refactored structure with clear boundaries. Use when users share large, mixed-concern, or hard-to-test components, or ask about splitting, refactoring, or improving component architecture.
3