MCP Server Evaluation Guide

Overview

This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.

Quick Reference

Evaluation Requirements

• Create 10 human-readable questions
• Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE
• Each question requires multiple tool calls (potentially dozens)
• Answers must be single, verifiable values
• Answers must be STABLE (won't change over time)