verifiable-science-envsBLOCK
Verifiable-reward RL environments and evals for scientific domains, starting with HLA/immunogenetics
Overview
From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.
Deterministic, executable-oracle RL environments and evaluation suites for clinical genomics — starting with HLA/immunogenetics.
Every answer is computed from the pinned IPD-IMGT/HLA release's own files. No human labels, no frequency data, no licensed tables — so the grader is auditable line-by-line, the sealed split regenerates on every release, and a model cannot have memorized the post-cutoff tasks.
The benchmarks
Working on this repo? Read `CLAUDE.md` first: a push to `main` deploys production, and the project's status, decisions and runbook live in the private portfolio hub rather than here.
Headline findings so far: every model family tested (Claude, Qwen, Mistral, Llama, Phi, Gemma) scores 0% on 2-field ambiguity expansion (0 of 30 tasks per model on the full 550-task suite), the core clinical trap; models fabricate allele names at 0.06–0.20 per task, and the anthropic/claude-sonnet-4-6 figure is 0.11 per task (54 of 550 tasks with a hallucinated name, clean re-run, all 550 tasks answered); on matching, the naive string baseline falls from 28% (family A) to 0%, and open models reach 0–14% because they count matched loci instead of chromosomes. Full tables with Wilson CIs on the bench pages; current state in the bench pages below.
[](https://colab.research.google.com/github/jasonbrelsford/verifiable-science-envs/blob/main/bench/reproduce.ipynb) Every headline figure above is
3616763edbf3OBSERVED · 2026-10-11Connect
Built from this server's own package name, version and transport as found in its source — not copied from anyone's documentation, so it cannot drift against a page we do not control. Replace the environment placeholders with a token scoped to the least it needs.
claude mcp add hlaverify-edge --env ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} --env GOOGLE_API_KEY=${GOOGLE_API_KEY} --env HF_TOKEN=${HF_TOKEN} --env HLA_VERIFY_API_KEYS=${HLA_VERIFY_API_KEYS} -- npx -y hlaverify-edge{
"mcpServers": {
"hlaverify-edge": {
"command": "npx",
"args": [
"-y",
"hlaverify-edge"
],
"env": {
"ANTHROPIC_API_KEY": "${ANTHROPIC_API_KEY}",
"GOOGLE_API_KEY": "${GOOGLE_API_KEY}",
"HF_TOKEN": "${HF_TOKEN}",
"HLA_VERIFY_API_KEYS": "${HLA_VERIFY_API_KEYS}"
}
}
}
}Exposed tools (11)
8 read · 3 write · 0 destructive.
| Tool | Risk | Description |
|---|---|---|
X-API-Key | read | Optional. Without a key /mcp shares the free tier (100 calls per UTC day per IP, 60 requests/minute); keys: https://api.hlaverify.com/pricing |
about | write | What this server is and is not, what to send it, benchmark evidence for why to use it, the beta state, and terms. |
allele_info | read | Look up one name in the pinned release and return what it is: |
beta_signup | write | Put a user on the free public beta |
check_typing | read | QC-check one HLA typing (all loci) against the pinned release: resolves every |
donor_compat | read | Donor/recipient immunogenetic compatibility under two published rule sets: HLA-B |
match_score | read | Count a donor-recipient HLA match by the published counting rules (R1-R6): allele |
normalize_allele | read | Normalize one reported HLA allele name (any era or reporting shorthand: legacy A*0101, Cw*0702 or Cw*07:02, |
research_access | write | Apply for free HLA-Verify access for an academic or nonprofit lab. |
validate_gl_string | read | Validate and normalize a GL String (Genotype List, ^ | + ~ / grammar): resolves |
verify_text | read | Scan HLA typing report text, or model output about HLA, for allele-shaped tokens and |
Trust audit
BLOCKgrade D · trust 69/100 Do not install this without reading the findings. The audit found something that could harm you or your machine.
| Layer | What it checks | Result |
|---|---|---|
| L0 | Provenance & inventory | PASS |
| L1 | Static analysis of the code | FAIL |
| L2 | Instruction surface (what it tells the agent) | PASS |
| L3 | Class-specific surface | PASS |
| L4 | Behavioural (sandbox) | SKIPPED |
What the source does
- Filesystem
- declared (1 observation(s))
- Network
- declared (3 observation(s))
- Shell
- none-observed
- Dependencies
- not all pinned
- Secrets in source
- none-found
Findings (22)
".env" ... urllib.request
verify = False
verify = False
return random.Random(int(hashlib.sha256(key.encode()).hexdigest()[:16], 16))
dependencies = ["verifiers>=0.3", "datasets", "sci-envs @ git+https://github.com/jasonbrelsford/verifiable-science-envs.git"]
.hf-publish
.hf-space
.tower-queue.json
h = hashlib.md5()
h = hashlib.md5()
h = hashlib.md5()
node <path>/mcpclient/probe.mjs http://127.0.0.1:<port>/mcp
assert.equal((await (await call(env, "GET", "/.well-known/oauth-authorization-server", { origin: "http://127.0.0.1:8796" })).json()).issuer, "http://127.0.0.1:8796");for (const id of ["http://agent.example.com/c.json", "https://127.0.0.1/c.json", "https://[::1]/c.json", "https://localhost/c.json",
const client = await registerClient(env, ["http://127.0.0.1:3000/callback", "http://localhost:4000/cb"]);
const ok = await openConsent(env, { client_id: client.client_id, redirect_uri: "http://127.0.0.1:51234/callback", code_challenge: challenge });verify = False
const bytes = Uint8Array.from(atob(v.slice(9, -2)), (c) => c.charCodeAt(0));
return Uint8Array.from(atob(s.replace(/-/g, "+").replace(/_/g, "/")), (c) => c.charCodeAt(0));
wrangler
<script type="application/ld+json">{"@context":"https://schema.org","@type":"SoftwareApplication","name":"HLA-Verify","applicationCategory":"Bioinformatics","operatingSystem":"Web, Python","url":"httpedge/test/fixtures.json
Gates applied: no_behavioural_pass.
3616763edbf3full audit observations/trust-audit/mcp-server/jasonbrelsford__verifiable-science-envs.json · Report an issue / request a re-scanAudit history
Every audit this server has had. A grade with a past is a grade somebody is still checking.
| Date | Source | Verdict | Grade | Score | Change |
|---|---|---|---|---|---|
| 2026-10-11 | 3616763edbf3 | BLOCK | D | 69 | first audit |
Questions
What is the verifiable-science-envs MCP server?
Verifiable-reward RL environments and evals for scientific domains, starting with HLA/immunogenetics
What tools does verifiable-science-envs expose?
11 in total: 8 read-only, 3 that write, and 0 that can delete or overwrite. Every one is listed on this page with its risk.
Is verifiable-science-envs safe to connect to an agent?
No — not without reading the findings first. The audit graded it D (69/100) and found 3 critical or high issues in the source. Each one is listed on this page with the file and line it is on.
What credentials does verifiable-science-envs need?
It reads ANTHROPIC_API_KEY, GOOGLE_API_KEY, HF_TOKEN, HLA_VERIFY_API_KEYS, OPENAI_API_KEY and STRIPE_API_KEY from the environment. Give it a token scoped to the least it needs — an agent that can be talked into calling a tool can be talked into calling it with your credentials.
How does verifiable-science-envs run?
It speaks streamable-http, so it runs as a service you connect to over the network. It is published on npm as hlaverify-edge.
How current is this page?
The grade is for one exact copy of the source (3616763edbf3), read on 2026-10-11. The repository is watched and re-audited when it changes.