Atlas / MCP servers / cavi-ai / mcp-eval

mcp-evalSAFE

Name: mcp-eval
Author: cavi-ai

mcp/cavi-ai/mcp-eval

MCP server by cavi-ai · agent evals · agents · ai · Audited 2026-10-11

🧐 Privacy-preserving MCP server evaluation: deterministic CI probe battery (18 probe kinds, readiness score, SARIF) plus production friction capture with findings that verify and close

Verdict

SAFE

Grade

Trust score

89 /100

Exposed tools

1 1r · 0w · 0d

Transport

stdio · streamable-http

License

MIT

Stars

Overview

From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.

[](LICENSE)

mcp-eval is an agnostic evaluation tool for MCP servers, with two complementary tracks:

Deterministic benchmark battery — run a strict, manifest-driven probe

suite against any MCP server (stdio or Streamable HTTP) and get a pass/fail verdict with fixed failure reasons. No LLM in the loop, no network services, fully reproducible: same server, same manifest, same result. Built for CI gating while you develop a server.

Friction capture — run your server behind a privacy-preserving shim in

production-like agent sessions, index what it actually costs agents, and promote repeated friction into queryable findings a development agent can act on. Findings close only after three consecutive green verification runs and reopen automatically on regression.

Capture journals and probe summaries omit raw payloads, error prose, and credentials. Store records under /store/ are eligible for sharing after reviewing annotation notes and the remaining files. Manifests, corpus target files, and retained corpus evidence are private artifacts outside that boundary; see Privacy and authorization.

Documentation: cavi-ai.xyz/docs/mcp-eval.

Install

brew install cavi-ai/tap/mcpeval
npm install --global @cavi-ai/mcp-eval
cargo install mcpeval --locked

Each installs mcpeval and the demo server mcpeval-demo. Homebrew (macOS and Linux, x64/arm64) and npm (also Windows x64) install the prebuilt release archive after checking its pinned SHA-256; the archives and their checksums are also on the releases page. Building from a checkout works too:

cargo build --release

The binaries are target/release/mcpeval and target/release/mcpeval-demo (.exe on Windows).

Quick

Read from source at commit fce0d5110a8eOBSERVED · 2026-10-11

Connect

Built from this server's own package name, version and transport as found in its source — not copied from anyone's documentation, so it cannot drift against a page we do not control.

claude-code

claude mcp add mcp-eval -- npx -y @cavi-ai/[email protected]

claude-desktop

{
  "mcpServers": {
    "mcp-eval": {
      "command": "npx",
      "args": [
        "-y",
        "@cavi-ai/[email protected]"
      ]
    }
  }
}

Exposed tools (1)

1 read · 0 write · 0 destructive.

Tool	Risk	Description
`read_state`	read	Read the fixture dataset

Trust audit

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

Layer	What it checks	Result
L0	Provenance & inventory	PASS
L1	Static analysis of the code	PASS
L2	Instruction surface (what it tells the agent)	PASS
L3	Class-specific surface	PASS
L4	Behavioural (sandbox)	SKIPPED

What the source does

Filesystem: declared (1 observation(s))
Network: declared (6 observation(s))
Shell: declared (1 observation(s))
Dependencies: pinned
Secrets in source: none-found

Findings (12)

LOWInventory / provenance · inv.hidden_file · CWE-1104

.pr-gates

.pr-gates

Why it matters. hidden member outside the usual dotfiles

Fix. review its purpose

LOWFilesystem / path · fs.traversal · CWE-22, CWE-59

actions/corpus-evidence/index.mjs:2

import { checkEvidence } from "../../scripts/corpus/check-evidence.mjs";

LOWFilesystem / path · fs.traversal · CWE-22, CWE-59

scripts/corpus/contract.test.mjs:7

const corpusFixture = JSON.parse(await readFile(new URL("../../tests/fixtures/corpus-v3.json", import.meta.url), "utf8"));

LOWFilesystem / path · fs.traversal · CWE-22, CWE-59

scripts/corpus/contract.test.mjs:8

const metadataCases = JSON.parse(await readFile(new URL("../../tests/fixtures/corpus-v3-metadata-cases.json", import.meta.url), "utf8"));

LOWFilesystem / path · fs.traversal · CWE-22, CWE-59

scripts/corpus/deployment.test.mjs:50

for (const change of [{ root: "relative" }, { sha256: "unknown" }, { executable: "../outside" }, { executable: "/outside" }, { entrypoint: "a/../../outside" }, { executable: "runtime\\outside" }]) {

LOWFilesystem / path · fs.traversal · CWE-22, CWE-59

scripts/distribution/publish-auth.test.mjs:11

new URL("../../.github/workflows/publish.yml", import.meta.url),