Arxiv Search

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkDeclared2 observations
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass, no license.

.claude/skills

Audit 0.4.1 · source sha 81994516a430 · 19 Oct 2026Report an issue or request a re-scan

• PublisherWILLOSCAR GitHub userOwner of the repository this skill is read from
• Source folderWILLOSCAR/research-units-pipeline-skills/.codex/skills/arxiv-searchThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict81994516a430A new audit runs when the source changes
• Repository statusNot archivedLast push 14 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Retrieve arXiv paper metadata with keyword queries or import an offline arXiv export, and save results as JSONL (`papers/papers_raw.jsonl`).

---
name: arxiv-search
description: "Retrieve arXiv paper metadata with keyword queries or import an offline arXiv export, and save results as JSONL (`papers/papers_raw.jsonl`)."
---

# arXiv Search (metadata-first)

## Triggers & routing

- **Trigger**: arXiv, arxiv, paper search, metadata retrieval, 文献检索, 论文检索, 拉取元数据, 离线导入.
- **Use when**: 需要一个初始论文集合(survey/snapshot 的 Stage C1),来源为 arXiv(在线检索或离线导入 export)。


Collect an initial paper set with enough metadata to support downstream ranking, taxonomy building, and citation generation.

When online, prefer rich arXiv metadata (categories, arxiv_id, pdf_url, published/updated, etc.). When offline, accept an export and convert it cleanly.

## Load Order

Always read:
- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):
- `assets/domain_packs/llm_agents.json` — pinned IDs, query rewrite rules for LLM agent topics

## Script Boundary

Use `scripts/run.py` only for:
- arXiv API retrieval and XML parsing
- offline export conversion (CSV/JSON/JSONL normalization)
- metadata enrichment via `id_list` backfill

Do not treat `run.py` as the place for:
- hardcoded topic detection or query rewriting (use domain packs)
- domain-specific pinned paper lists (externalize to `assets/domain_packs/`)

## Contract-driven behavior

- Domain-pack query rewriting is the default for broad discovery Workflows.
- A focused Workflow may set
  `quality_contract.retrieval_policy.domain_pack_query_mode: explicit`; in that
  mode, the query list in `queries.md` remains authoritative and the domain
  pack must not replace its topic focus.
- `quality_contract.retrieval_policy.minimum_records` turns a Workflow's raw
  candidate-pool floor into a strict quality-gate check.

## Input

- `queries.md` (keywords, excludes, time window)

## Outputs

- `papers/papers_raw.jsonl` (JSONL; 1 paper per line)
  - Each record includes at least: `title`, `authors`, `year`, `url`, `abstract`
  - When using the arXiv API online mode, records also include helpful metadata: `arxiv_id`, `pdf_url`, `categories`, `primary_category`, `published`, `updated`, `doi`, `journal_ref`, `comment`
- Convenience index (optional but generated by the script):
  - `papers/papers_raw.csv`

## Decision: online vs offline

- If you have network access: run arXiv API retrieval.
- If not: import an export the user provides (CSV/JSON/JSONL) and normalize fields.
- Hybrid: if you import offline but still have network later, you can **enrich missing fields** (abstract/authors/categories) via arXiv `id_list` using `--enrich-metadata` or `queries.md` `enrich_metadata: true`.

## Workflow (heuristic)

1. Read `queries.md` and expand into concrete query strings.
2. Retrieve results (online) or import an export (offline).
3. Normalize every record to include at least:
   - `title`, `authors` (array), `year`, `url`, `abstract`
4. Keep the set broad at this stage; dedupe/ranking comes next.
5. Apply time window and `max_results` if specified.

## Quality checklist

- [ ] `papers/papers_raw.jsonl` exists.
- [ ] Each line is valid JSON and contains `title`, `authors`, `year`, `url`.

## Side effects

- Allowed: create/overwrite `papers/papers_raw.jsonl`; append notes to `STATUS.md`.
- Not allowed: write prose sections in `output/` before writing is approved.

## Script

### Quick Start

- `uv run python .codex/skills/arxiv-search/scripts/run.py --help`
- Online: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "<query>" --max-results 200`
- Offline import: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --input <export.csv|json|jsonl>`

### All Options

- `--query <q>`: repeatable; multiple queries are unioned
- `--exclude <term>`: repeatable; excludes applied after retrieval
- `--max-results <n>`: cap total retrieved
- `--input <export.*>`: offline mode (CSV/JSON/JSONL)
- `--enrich-metadata`: best-effort enrich via arXiv `id_list` (needs network)
- `queries.md` also supports: `keywords`, `exclude`, `time window`, `max_results`, `enrich_metadata`

### Examples

- Online (multi-query + excludes):
  - `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query "LLM agent" --query "tool use" --exclude "survey" --max-results 300`
- Fetch a single paper by arXiv ID (direct `id_list` fetch):
  - `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace> --query 2509.02547 --max-results 1`
- Offline auto-detect (no flags):
  - Place `papers/import.csv` (or `.json/.jsonl`) under the workspace, then run: `uv run python .codex/skills/arxiv-search/scripts/run.py --workspace <workspace>`
- Offline import + time window (via `queries.md`):
  - Set `- time window: { from: 2022, to: 2025 }` then run offline import normally

## Troubleshooting

### Common Issues

#### Issue: `papers/papers_raw.jsonl` is empty

**Symptom**:
- Script exits with “No results returned ...” or output file is empty.

**Causes**:
- Network is blocked (online mode).
- Queries are too narrow or `queries.md` is empty.

**Solutions**:
- Use offline import: place `papers/import.csv|json|jsonl` in the workspace or pass `--input`.
- Broaden keywords and reduce excludes in `queries.md`.
- Run with explicit `--query` to sanity-check the parser.

#### Issue: Offline import records miss fields

**Symptom**:
- Downstream steps fail because records miss `authors/year/abstract/url`.

**Causes**:
- Export columns don’t match expected fields; upstream export is incomplete.

**Solutions**:
- Ensure the export contains at least `title`, `authors`, `year`, `url`, `abstract`.
- If you later have network, use `--enrich-metadata` to backfill missing fields (best effort).

### Recovery Checklist

- [ ] Confirm `queries.md` has non-empty `keywords` (or pass `--query`).
- [ ] If offline: confirm workspace has `papers/import.*` and rerun.
- [ ] Spot-check 3–5 JSONL lines: valid JSON + required field

Excerpt, first 6,000 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

1. 2026-10-19
   ✓SAFE B 89 / 100

   first audit · source 81994516a430

Questions about Arxiv Search

• github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.c — the audited source at commit 81994516a430 and every finding aboveObserved · 19 Oct 2026

Spotted something wrong? Report a correction · Methodology