Literature Engineer

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkDeclared5 observations
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass, no license.

.claude/skills

Audit 0.4.1 · source sha 81994516a430 · 19 Oct 2026Report an issue or request a re-scan

• PublisherWILLOSCAR GitHub userOwner of the repository this skill is read from
• Source folderWILLOSCAR/research-units-pipeline-skills/.codex/skills/literature-engineerThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict81994516a430A new audit runs when the source changes
• Repository statusNot archivedLast push 14 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: Multi-route literature expansion + metadata normalization for evidence-first surveys.

---
name: literature-engineer
description: "Multi-route literature expansion + metadata normalization for evidence-first surveys."
---

# Literature Engineer (evidence collector)

## Triggers & routing

- **Trigger**: evidence collector, literature engineer, 文献扩充, 多路召回, snowballing, cited by, references, 元信息增强, provenance.
- **Use when**: Workflow 需要按锁定的 retrieval policy 扩充候选文献并补齐可追溯 metadata。


Goal: build a **large, verifiable candidate pool** for downstream dedupe/rank, mapping, notes, citations, and drafting.

This skill is intentionally **evidence-first**: if you can't reach the target size with verifiable IDs/provenance, the correct behavior is to **block** and ask for more exports / enable network, not to fabricate.

## Load Order

Always read:
- `references/domain_pack_overview.md` — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):
- `assets/domain_packs/llm_agents.json` — pinned classic/survey arXiv IDs for LLM agent topics

## Script Boundary

Use `scripts/run.py` only for:
- multi-route offline import, normalization, and provenance tagging
- online arXiv/Semantic Scholar API retrieval
- snowball expansion and deduplication
- retrieval report generation

Do not treat `run.py` as the place for:
- hardcoded pinned arXiv ID lists (use domain packs)
- hardcoded topic detection logic (use domain packs)

## Inputs

- `queries.md`
  - `keywords`, `exclude`, `max_results`, `time window`
- Optional offline sources (any combination; all are merged):
  - `papers/import.(csv|json|jsonl|bib)`
  - `papers/arxiv_export.(csv|json|jsonl|bib)`
  - `papers/imports/*.(csv|json|jsonl|bib)`
- Optional snowball exports (offline):
  - `papers/snowball/*.(csv|json|jsonl|bib)`

## Outputs

- `papers/papers_raw.jsonl`
  - 1 record per line; minimum fields:
    - `title` (str), `authors` (list[str]), `year` (int|""), `url` (str)
    - stable identifier(s): `arxiv_id` and/or `doi`
    - `abstract` (str; may be empty in offline mode)
    - `source` (str) + `provenance` (list[dict])
- `papers/papers_raw.csv` (human scan)
- `papers/retrieval_report.md` (route counts, missing-meta stats, next actions)

## Workflow (multi-route)

1. **Offline-first merge**: ingest all available offline exports (and label provenance per file).
2. **Online retrieval (optional)**: if enabled, run arXiv API retrieval for each keyword query.
3. **Snowballing (optional)**: expand from seed papers via references/cited-by (online), or merge offline snowball exports.
4. **Normalize + dedupe**: canonicalize IDs/URLs, merge duplicates while unioning `provenance`.
5. **Report**: write a concise retrieval report with coverage buckets and missing-meta counts.

## Quality checklist

- [ ] Candidate pool meets the active Workflow's declared retrieval floor **without fabrication**. For Workflows with `retrieval_policy.minimum_records`, use that value; survey profiles may instead derive a stricter pool target from `core_size`.
- [ ] Each record has a stable identifier (`arxiv_id` or `doi`, plus `url`).
- [ ] Each record has provenance: which route/file/API produced it.

## Script

### Quick Start

- `uv run python .codex/skills/literature-engineer/scripts/run.py --help`


### All Options

- See `uv run python .codex/skills/literature-engineer/scripts/run.py --help`.
- Reads retrieval config from `queries.md`.
- Offline inputs (merged if present): `papers/import.(csv|json|jsonl|bib)`, `papers/arxiv_export.(csv|json|jsonl|bib)`, `papers/imports/*.(csv|json|jsonl|bib)`.
- Optional offline snowball inputs: `papers/snowball/*.(csv|json|jsonl|bib)`.
- Online expansion requires network: use `--online` and/or `--snowball`.
- Online retrieval is best-effort: arXiv API can be flaky in some environments; the script will also attempt a Semantic Scholar route when needed.
- For LLM-agent topics, the script also performs a best-effort **pinned arXiv id_list fetch** (canonical classics like ReAct/Toolformer/Reflexion/Voyager/Tree-of-Thoughts + a small prior-survey seed set) so `ref.bib` can include must-cite anchors even when keyword search misses them.
- If HTTPS/TLS to external domains is unstable, the Semantic Scholar route is fetched via the `r.jina.ai` proxy so the pipeline can still self-boot without manual exports.
- When an online run returns `0` records due to transient network errors, a simple rerun is often sufficient (the pipeline should not fabricate).


### Examples

- Offline imports only:
  - Put exports under `papers/imports/` then run:
    - `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace>`

- Explicit offline inputs (multi-route):
  - `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --input path/to/a.bib --input path/to/b.jsonl`

- Online arXiv retrieval (needs network):
  - `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --online`

- Snowballing (needs network unless you provide offline snowball exports):
  - `uv run python .codex/skills/literature-engineer/scripts/run.py --workspace <workspace> --snowball`

## Troubleshooting

### Issue: cannot reach the active Workflow's candidate-pool target

**Symptom**:
- `papers/papers_raw.jsonl` is below the explicit or profile-derived minimum declared by the locked Workflow.

**Causes**:
- Only a small offline export was provided.
- Network is blocked so online retrieval/snowballing can't run.

**Solutions**:
- Provide additional exports under `papers/imports/` (multiple routes/queries).
- Provide snowball exports under `papers/snowball/`.
- Enable network and rerun with `--online --snowball`.

### Issue: many records missing stable IDs

**Symptom**:
- Report shows many entries with empty `arxiv_id` and `doi`.

**Solutions**:
- Prefer arXiv/OpenReview/ACL exports that include stable IDs.
- If you have network, rerun with `--online` to backfill arXiv IDs.
- Filter out ID-less entries before downstream citation generation.

Excerpt, first 5,973 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

1. 2026-10-19
   ✓SAFE B 89 / 100

   first audit · source 81994516a430

Questions about Literature Engineer

• github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.c — the audited source at commit 81994516a430 and every finding aboveObserved · 19 Oct 2026

Spotted something wrong? Report a correction · Methodology