Atlas · Audit version 0.4.1 · Latest audit 29 Oct 2026
How we verify what we publish
Evidence over marketing. Every price, spec and verdict on ModelsAtlas carries its source, the date we checked it, and what kind of fact it is.
Cite
Cite this page
ModelsAtlas. “How we verify what we publish”. Retrieved 30 Oct 2026. https://modelsatlas.com/methodology
Four kinds of fact
Shown next to any value you open
| Class | Meaning | Example |
|---|---|---|
| Asserted | The vendor says so on its own page or in its documentation. We link that page. | A model's list price on its maker's pricing page |
| Observed | We measured or read it ourselves, from our catalogue, a repository or a pricing page. | A repository's star count; the files an audit read; a tool's plan price |
| Derived | Computed from other facts, never typed. | A grid reference; a verdict split; a monthly cost estimate |
| Editorial | Our judgement, labelled as such. | A “choose A if…” line; a best-for ranking |
Models, providers and prices
Model records come from our catalogue, checked against each vendor's own pricing and documentation pages. Every price shows the date it was retrieved, and opening it shows the source. Vendor documentation is named and linked, because it is the vendor speaking; where a figure comes from our own gathering it reads “Our catalogue”.
Prices never sit on a model: they sit on an offer, one host serving one model at one tier (standard, batch, free and so on). That is why a model page can show several prices for the same model. A model's grid reference is derived from its standard-tier input price and its context length; when either is missing it has no grid reference.
A tool's plan prices are shown only after we have read the vendor's own pricing page. Tools we have not read are marked “Not yet checked” and carry no price. Model identifiers never change: when a vendor renames a model we add a record and redirect the old address.
The trust audit for skills and MCP servers
Every listing is read in full at a named commit
| Layer | What we check | Results across 19,992 listings |
|---|---|---|
L0 Provenance | Who owns the repository, its licence, activity and archived status, and whether the package matches the source. | PASS 14,735 WARN 4,981 FAIL 276 |
L1 Source scan | Filesystem, network and shell use, hard-coded secrets and unpinned dependencies. | PASS 5,704 WARN 1,620 FAIL 1,912 NA 10,756 |
L2 Instructions | Prompt-injection patterns and authority framing in the skill, the README and tool descriptions. | PASS 18,157 WARN 284 FAIL 1,551 |
L3 Inventory | Which tools an MCP server exposes, each classed read, write or destructive. | PASS 15,610 WARN 4,382 |
L4 Sandbox | Runs the package isolated and records what it actually touches. | SKIPPED 19,992 |
Findings in tests, examples or build scripts are shown but marked outside shipped code. The sandbox layer (L4) is not running yet, so every listing shows it as skipped.
Verdicts and grades
The audit found nothing in the source that contradicts what the listing says it does. Install it as you would any dependency.
Medium findings, high findings that stop short of a block, or broad permissions such as shell access or destructive tools.
A critical finding, instructions that try to override the agent, data sent to an undeclared host, or high-severity findings in the source. The page stays public and shows why.
The grade (B to F today) follows the score, which runs from 0 to 100; B sits in the 80s, C in the 70s, D in the 60s and F at about 60 and below. The verdict is separate: it also reflects what was found, so a listing can be CAUTION with grade B. Grade A is held back for packages that have also passed a sandboxed behavioural run, which is why none has it yet.
What goes into the score
Audit version 0.4.1
Each audit scores 8 dimensions from 0 to 100 and combines them into the overall score. The verdict follows the findings, never popularity, stars or who published it. Scores published by others are cited on the page but never blended into ours.
Verify our results yourself
Every verdict names the commit we read
- Check out the exact commit.
Each audit records the commit it read, for example
743adeacb086for mattpocock/skills. Open the repository at that commit, not the branch head. - Open the file and line.
Each finding names the rule that fired, the file and the line, with the excerpt. Rules are named like
net.raw_iporcode.eval_exec, so you can search the source for the same thing. - Compare with the page.
The page shows the verdict, grade, score and every finding for that commit. If the source has moved on, the page shows the audits it has had so far.
- Disagree? Tell us.
Report the finding with your evidence from the page's own link. A correction is a public GitHub issue, so what was asked and what was decided stays visible.
Corrections
Every page ends with its sources and a Report a correction link. Send a source URL for the right value. We do not accept payment for listings, placement or verdicts: how we make money says what we do earn from.
Questions
What is the difference between a verdict and a grade?
Why does no listing have grade A?
Can a publisher remove a finding?
Keep going in your AI
Opens with this page as the source. We add nothing else to the prompt.
- What is the difference between a verdict and a grade?
- Why does no listing have grade A?
- Can a publisher remove a finding?