crawlerSAFE
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
Overview
From the repository's own README, as read at the audited commit. Badges and raw HTML are left out.
This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.
polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.
- The package:
packages/polite-crawleris on npm as `polite-crawler`. - The site:
packages/siteis the documentation, with interactive demos that use the real code: .
npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler npx polite-crawler find "robots exclusion" claude mcp add polite-crawler -- npx -y polite-crawler mcp
Development
The repository is a pnpm workspace. It needs Node.js 22.13 or later.
pnpm install pnpm typecheck pnpm test pnpm lint:ste pnpm --filter polite-crawler build
The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter `ste-lint` examines it in CI.
License
MIT
f862f30c4440OBSERVED · 2026-10-12Connect
Built from this server's own package name, version and transport as found in its source — not copied from anyone's documentation, so it cannot drift against a page we do not control.
claude mcp add site -- npx -y @crawler/[email protected]
{
"mcpServers": {
"site": {
"command": "npx",
"args": [
"-y",
"@crawler/[email protected]"
]
}
}
}Exposed tools (4)
4 read · 0 write · 0 destructive.
| Tool | Risk | Description |
|---|---|---|
crawl_fetch | read | |
crawl_find | read | |
crawl_read | read | |
crawl_verify | read |
Trust audit
SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.
| Layer | What it checks | Result |
|---|---|---|
| L0 | Provenance & inventory | PASS |
| L1 | Static analysis of the code | PASS |
| L2 | Instruction surface (what it tells the agent) | PASS |
| L3 | Class-specific surface | PASS |
| L4 | Behavioural (sandbox) | SKIPPED |
What the source does
- Filesystem
- declared (2 observation(s))
- Network
- declared (2 observation(s))
- Shell
- declared (3 observation(s))
- Dependencies
- not all pinned
- Secrets in source
- none-found
Findings (7)
import { packagesDir, repoRoot } from "../../paths.ts";import { packagesDir, repoRoot } from "../../paths.ts";base = `http://127.0.0.1:${String((server.address() as AddressInfo).port)}`;@mark1russell7/cue, @mark1russell7/ste-lint, tsx, typescript
@clack/prompts, @mark1russell7/cue, @types/node, typescript, vitest
@modelcontextprotocol/sdk, @mozilla/readability, linkedom, undici, unpdf, zod, @mark1russell7/cue, @types/node
@astrojs/react, @astrojs/starlight, @fontsource-variable/atkinson-hyperlegible-mono, @fontsource-variable/atkinson-hyperlegible-next, astro, react, react-dom, @mark1russell7/cue
Gates applied: no_behavioural_pass.
f862f30c4440full audit observations/trust-audit/mcp-server/mark1russell7__crawler.json · Report an issue / request a re-scanAudit history
Every audit this server has had. A grade with a past is a grade somebody is still checking.
| Date | Source | Verdict | Grade | Score | Change |
|---|---|---|---|---|---|
| 2026-10-12 | f862f30c4440 | SAFE | B | 89 | first audit |
Questions
What is the crawler MCP server?
polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.
What tools does crawler expose?
4 in total: 4 read-only, 0 that write, and 0 that can delete or overwrite. Every one is listed on this page with its risk.
Is crawler safe to connect to an agent?
The audit found nothing in the source that contradicts what it says it does, and graded it B (89/100). Grade A is held back for packages that have also passed a sandboxed behavioural run, which is why a clean server reads B.
What credentials does crawler need?
No credential environment variables were found in its source, so it appears to need none.
How does crawler run?
It speaks stdio, so it runs as a local process your client starts. It is published on npm as @crawler/site at 0.1.0.
How current is this page?
The grade is for one exact copy of the source (f862f30c4440), read on 2026-10-12. The repository is watched and re-audited when it changes.