Rlm EvalSAFE
turns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then runs tree search with parallel subagents.
Overview
From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.
Harness for measuring whether evo's optimize skill actually helps an orchestrator find cross-cutting patterns in an experiment tree. Named after the Recursive Language Model idea (Zhang/Kraska/Khattab, arXiv:2512.24601) that seeded it -- the skill's hard rule is to fan out across experiments via parallel sub-agents rather than read every trace inline.
It runs paired trials under claude -p:
- V (vanilla): the pre-RLM
SKILL.md, snapshotted asbaseline_SKILL.md. - R (current): whatever is live at
plugins/evo/skills/optimize/SKILL.md.
Each trial gets a synthetic .evo/ tree with six planted patterns (A/B/C/D/E/S) and is scored both strictly (planted-only) and by an LLM judge that credits legitimate grounded observations beyond the planted set.
Files
rlm_eval.py-- CLI:check,setup,trial,matrix,score,clean.generate_fixture.py-- case-spec + one batched LLM call per case; writes
.evo/run_0001/experiments//... and ground_truth.json.
score.py-- strict scorer. Credits only planted patterns.score_llm.py--claude -pjudge with 0/1/2 rubric +planted_recall.baseline_SKILL.md-- frozen V skill. Do not edit; replace only when
intentionally re-baselining.
fixtures/-- generated, gitignored.
Trial outputs default to trials/ at repo root, also gitignored.
Dependencies
claudeCLI on PATH.- Python 3.10+.
- macOS or Linux for sandboxed Bash isolation (Seatbelt / bubblewrap). On
native Windows the harness falls back to permission deny rules only, which blocks Claude's Read/Edit/Grep but not arbitrary Bash subprocesses -- use WSL2 for full isolation.
Run python scripts/rlm_eval/rlm_eval.py check to verify.
Typical run
cd scripts/rlm_eval # One-time: generate small/medium/large fixtures. python rlm_eval.py setup --seed 1 # Single trial against R (default), large fixture. python rlm_eval.py trial \ --fixture fixtures/large \ --out ../../trials/s
f8edbd9e89c5OBSERVED · 2026-10-12Host compatibility
What the documentation claims. We have not run a compatibility test.
| Host | Status | Notes |
|---|---|---|
| claude-code | mentioned | |
| codex | mentioned |
Trust audit
SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.
| Layer | What it checks | Result |
|---|---|---|
| L0 | Provenance & inventory | PASS |
| L1 | Static analysis of the code | PASS |
| L2 | Instruction surface (what it tells the agent) | PASS |
| L3 | Class-specific surface | WARN |
| L4 | Behavioural (sandbox) | SKIPPED |
What the source does
- Filesystem
- declared (4 observation(s))
- Network
- none-observed
- Shell
- none-observed
- Dependencies
- pinned
- Secrets in source
- none-found
Findings (1)
Gates applied: no_behavioural_pass.
f8edbd9e89c5full audit observations/trust-audit/skill/evo-hq__rlm_eval.json · Report an issue / request a re-scanAudit history
Every audit this skill has had.
| Date | Source | Verdict | Grade | Score | Change |
|---|---|---|---|---|---|
| 2026-10-12 | f8edbd9e89c5 | SAFE | B | 89 | first audit |
Questions
What does the Rlm Eval skill do?
turns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then runs tree search with parallel subagents.
Is Rlm Eval safe to install?
The audit found nothing in the source that contradicts what it says it does, and graded it B (89/100). Grade A is held back for packages that have also passed a sandboxed behavioural run, which is why a clean skill reads B.
What can Rlm Eval access on my machine?
The audit observed that it reads or writes files. Each of those is consistent with what it says it does. Secrets in the source: none found.
Which assistants does Rlm Eval work with?
Its documentation mentions claude-code and codex. That is what the text claims, not a compatibility test we ran.
How current is this page?
The grade is for one exact copy of the source (f8edbd9e89c5), read on 2026-10-12. The repository is watched, and a new audit runs when it changes — this is the first audit.