eval-audit

Malicious tool definition detected

Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards 2. Run diagnostic checks across six areas 3. Produce a findings report ordered by impact, with each finding linking to a fix ## Prerequisites Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure." ## Connecting to Eval Infrastructure Check whether the user has an observability

- **Feedback:** Use explicit customer feedback (complaints, thumbs-down signals) to filter traces. **Finding if insufficient:** Small datasets produce unreliable failure rates and wide confidence intervals.