Evaluation

Multi-layer evaluation covering protocol compliance, quality, security, and LLM-assisted usability.

Overview

MCP Lens runs four evaluation layers. The first three (Protocol, Quality, Security) are static analysis and require no external dependencies. The fourth (LLM) requires a configured LLM provider — see LLM Configuration.

Evaluation Layers

Layer Checks What it catches
Protocol 12 Missing names, invalid schemas, malformed annotations
Quality 11 Vague descriptions, missing parameter docs, naming inconsistencies
Security 6 Annotation mismatches, prompt injection, SQL injection, data exfiltration
LLM 5+ Description confusion, wrong tool selection, argument hallucination, overlap ambiguity, unsafe activation

How Tool Selection Works

  1. Scenario generation — for each tool, the LLM generates a realistic user request that should trigger that tool.
  2. Tool selection test — the scenario is sent to the LLM along with all tool definitions using native function-calling. The LLM picks which tool to use.
  3. Verdict — PASS if the correct tool is selected; WARN if a valid prerequisite tool is chosen; FAIL if there is genuine confusion.
Important: The LLM is only asked which tool to select. No tool is ever executed during evaluation.

LLM Checks Detail

Check What it does Why it matters
Description Clarity LLM rates description clarity on a 1–10 scale If the LLM can't understand the description, no agent can use it
Tool Selection Auto-generates a scenario and asks the LLM to pick the right tool Tests whether tools are distinguishable from each other
Argument Generation LLM generates arguments and validates them against the schema Catches unclear parameter schemas that lead to invalid inputs
Overlap Disambiguation For overlapping tool pairs, tests if the LLM can tell them apart Surfaces tools with similar descriptions that confuse agents
Safety Resistance Sends benign prompts and checks if the LLM avoids destructive tools Prevents accidental misuse of dangerous operations

Scoring

Each layer is scored 0–100 based on check results weighted by severity:

Severity Weight
Critical 5.0
High 3.0
Medium 2.0
Low 1.0
Info 0.5

The overall score is a weighted combination of all active layers. The gate threshold is 70.0. Critical failures in the protocol or security layers force gate failure regardless of the overall score.

False Positives

  • Mark any failure as a false positive with a justification
  • Persisted in report metadata and survive re-runs
  • Keyed by {layer}:{tool_name}:{check_id}
  • Displayed with strikethrough text and a purple FP badge in the UI

Export

Download tools and evaluation results as JSON, YAML, or PDF. Selectable sections include:

  • Tool overview
  • Parameter details
  • Raw schemas
  • Scores
  • Layer results
  • False positives