SkillEvaluatorBLOCK
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Overview
From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.
[](LICENSE) [](https://www.python.org/) [](https://docs.nvidia.com/skills/skillevaluator/) [](https://arxiv.org/abs/2608.20614) [](https://developer.nvidia.com/blog/evaluating-ai-agent-skill-performance-with-nvidia-skillevaluator/)
📺 Livestream: Ask the Experts: Evaluating Agent Skills
SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.
Agent skills extend AI agents with instructions and supporting files, as defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline. Skills that pass are published to the NVIDIA skills catalog.
Research foundation. Tier 3 implements methods from *Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills*: paired with/without-skill trials and Skill Lift; SkillEvaluator provides validation, deduplication, dataset authoring, and reporting.
Three-tier overview
Run each tier independently.
b59a4f805369OBSERVED · 2026-10-14Install
Commands as the repository documents them. They are shown, not run.
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
git clone https://github.com/NVIDIA/SkillEvaluator.git
uv sync --python 3.13 --extra all
uv run --extra all skillevaluator tier1 ./my-skill
uv run --extra all
npm install -g fern-api
Host compatibility
What the documentation claims. We have not run a compatibility test.
| Host | Status | Notes |
|---|---|---|
| claude-code | mentioned | |
| codex | mentioned | |
| cursor | mentioned |
What it tells the agent
The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.
--- name: simple description: Summarize short user notes into clear action items. metadata: author: Simple Skill Author <[email protected]> --- # Simple ## Instructions Read the user's note and return a concise list of action items. If the note has no clear action, say that no action item was found. ## Examples User: "Follow up with Maya tomorrow about the launch checklist." Assistant: "Action item: Follow up with Maya tomorrow about the launch checklist."
Trust audit
BLOCKgrade F · trust 54/100 Do not install this without reading the findings. The audit found something that could harm you or your machine.
| Layer | What it checks | Result |
|---|---|---|
| L0 | Provenance & inventory | PASS |
| L1 | Static analysis of the code | FAIL |
| L2 | Instruction surface (what it tells the agent) | PASS |
| L3 | Class-specific surface | PASS |
| L4 | Behavioural (sandbox) | SKIPPED |
What the source does
- Filesystem
- declared (15 observation(s))
- Network
- declared (17 observation(s))
- Shell
- declared (10 observation(s))
- Dependencies
- pinned
- Secrets in source
- found
Findings (25)
model = yaml.load( # nosec B506
add(f"""{PY} -c "exec(open('{SCRIPT}').read())" """.strip())help="Skip the without-skill baseline in live eval (no lift analysis, faster).",
- pattern: eval(...)
- pattern: exec(...)
exec(...)
"~/.ssh",
".netrc",
".git-credentials",
"id_ecdsa",
"id_ed25519",
re.compile(r"exfiltrat|leak .*secret|print .*token", re.IGNORECASE),
message="Agent issued a network command that could exfiltrate data",
re.compile(r"exfiltrat|leak .*secret|print .*token", re.IGNORECASE),
message="Agent issued a network command that could exfiltrate data",
expected_origin = f"http://127.0.0.1:{port}" if port is not None else ""origin = f"http://127.0.0.1:{server.server_port}"if relative_uri is not None and exc.args == (f"Invalid endpoint: http://169.254.170.2{relative_uri}",):secret = "real-sandbox-stream-secret"
api_key="test-readiness-worker-key",
api_key="test-repeated-cancel-key",
secret = "nonzero-stream-secret"
secret = "local-callback-base-error-secret"
"case-ghp_abcdefghijklmnopqrstuvwxyz0123456789",
.gitleaks.toml
Gates applied: no_behavioural_pass.
b59a4f805369full audit observations/trust-audit/skill/nvidia__skillevaluator.json · Report an issue / request a re-scanAudit history
Every audit this skill has had.
| Date | Source | Verdict | Grade | Score | Change |
|---|---|---|---|---|---|
| 2026-10-14 | b59a4f805369 | BLOCK | F | 54 | first audit |
Questions
What does the SkillEvaluator skill do?
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Is SkillEvaluator safe to install?
No — not without reading the findings first. The audit graded it F (54/100) and found 11 critical or high issues in the source. Each one is listed on this page with the file and line it is on.
What can SkillEvaluator access on my machine?
The audit observed that it reaches the network, runs shell commands and reads or writes files. Each of those is consistent with what it says it does. Secrets in the source: found — see the findings.
Which assistants does SkillEvaluator work with?
Its documentation mentions claude-code, codex and cursor. That is what the text claims, not a compatibility test we ran.
How current is this page?
The grade is for one exact copy of the source (b59a4f805369), read on 2026-10-14. The repository is watched, and a new audit runs when it changes — this is the first audit.