Arxiv Latex SourceSAFE

🔬 A curated collection of 23,000+ agent skills for empirical research across 8 social science disciplines. | 精选 23,000+ AI Agent 技能库,覆盖8大社会科学学科的实证研究。CoPaper.AI 20分钟完成一篇可复现的规范实证论文,并支持用户上传 Skills。-- Maintained by CoPaper.AI from Stanford REAP.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: arxiv-latex-source
description: "Download and parse LaTeX source files from arXiv preprints"
metadata:
  openclaw:
    emoji: "📜"
    category: "literature"
    subcategory: "fulltext"
    keywords: ["arXiv", "LaTeX source", "paper parsing", "formula extraction", "full text", "preprint"]
    source: "https://info.arxiv.org/help/bulk_data_s3.html"
---

# arXiv LaTeX Source Access Guide

## Overview

arXiv stores the original LaTeX source files for the vast majority of its 2.4 million+ preprints. Accessing LaTeX source provides major advantages over PDF parsing: exact mathematical notation as written by the author, structured sections and labels, machine-readable bibliography entries, and intact figure captions, table data, and cross-references.

For formula extraction, citation graph construction, section-level text analysis, or training data curation for scientific language models, LaTeX source is the gold standard. PDF parsing introduces OCR errors in equations, loses structural hierarchy, and mangles complex tables.

The e-print endpoint serves source bundles as gzip-compressed tarballs (`.tar.gz`) containing `.tex` files, figures, `.bib`/`.bbl` bibliography files, style files, and supplementary materials. No authentication is required.

## Authentication

No authentication or API key is required. The e-print endpoint is publicly accessible. However, arXiv asks that automated tools set a descriptive `User-Agent` header and comply with rate limits.

## Core Endpoints

### Download LaTeX Source

- **URL**: `GET https://arxiv.org/e-print/{arxiv_id}`
- **Response**: `application/gzip` — a `.tar.gz` archive containing the source files
- **Parameters**:
  | Param | Type | Required | Description |
  |-------|------|----------|-------------|
  | arxiv_id | string | Yes | arXiv identifier, e.g. `2301.00001` or `2301.00001v2` for a specific version |

- **Example**:
  ```bash
  # Download source archive (response: 200, application/gzip, ~1.3 MB)
  curl -sL -o source.tar.gz "https://arxiv.org/e-print/2301.00001"

  # List archive contents
  tar tz -f source.tar.gz | head -10
  # ACM-Reference-Format.bbx
  # ACM-Reference-Format.bst
  # Image_1.jpg
  # README.txt
  # acmart.cls
  ```

- **Content-Disposition header**: `attachment; filename="arXiv-2301.00001v1.tar.gz"`
- **ETag**: SHA-256 hash provided for caching: `sha256:f1ffe8ec...`

### Format Detection

The endpoint almost always returns a gzip-compressed tar archive. Rare cases (very old or single-file submissions) may return a single gzip-compressed `.tex` file without tar wrapper. Always verify format before extracting:

```bash
curl -sL "https://arxiv.org/e-print/{arxiv_id}" -o source.gz
file source.gz  # "gzip compressed data, was 'XXXX.tar', ..."
```

### Metadata API (Companion)

Pair source downloads with the arXiv Atom API for structured metadata:

- **URL**: `GET https://export.arxiv.org/api/query?id_list={arxiv_id}`
- **Response**: Atom XML with `<title>`, `<author>`, `<summary>`, `<category>`, `<published>`
- **Example**: `curl -s "https://export.arxiv.org/api/query?id_list=2301.00001"`

## LaTeX Source Parsing Guide

### Locating the Main .tex File

A source archive typically contains multiple files. To find the main document:

1. Look for `\documentclass` in `.tex` files — this marks the root document
2. Check for a `README.txt` that may specify the main file
3. If multiple `.tex` files contain `\documentclass`, prefer the one with `\begin{document}`

```python
import tarfile, re

def find_main_tex(tar_path):
    with tarfile.open(tar_path, 'r:gz') as tar:
        tex_files = [m for m in tar.getmembers() if m.name.endswith('.tex')]
        for member in tex_files:
            content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
            if r'\documentclass' in content and r'\begin{document}' in content:
                return member.name, content
    return None, None
```

### Extracting Sections

LaTeX sections follow a predictable hierarchy:

```python
import re

def extract_sections(tex_content):
    pattern = r'\\(section|subsection|subsubsection)\{([^}]+)\}'
    sections = re.findall(pattern, tex_content)
    return [(level, title) for level, title in sections]
    # [('section', 'Introduction'), ('section', 'Related Work'), ...]
```

### Extracting Equations

```python
def extract_equations(tex_content):
    patterns = [
        r'\\\[(.+?)\\\]',
        r'\\begin\{equation\}(.+?)\\end\{equation\}',
        r'\\begin\{align\*?\}(.+?)\\end\{align\*?\}',
    ]
    equations = []
    for pat in patterns:
        equations.extend(re.findall(pat, tex_content, re.DOTALL))
    return equations
```

### Extracting Bibliography

Parse `.bib` files (BibTeX entries) or `.bbl` files (compiled `\bibitem` commands):

```python
def extract_bibliography(tar_path):
    refs = []
    with tarfile.open(tar_path, 'r:gz') as tar:
        for member in tar.getmembers():
            if member.name.endswith('.bib'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'@\w+\{([^,]+),(.+?)\n\}', content, re.DOTALL))
            elif member.name.endswith('.bbl'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'\\bibitem.*?\{(.+?)\}', content))
    return refs
```

## Rate Limits

- **Maximum**: 4 requests per second for automated access
- **Recommended**: 1 request/second with delays between sequential downloads
- **Bulk access**: For 1000+ papers, use the arXiv S3 bulk data mirror instead
- **HTTP 429**: Rate limit exceeded; implement exponential backoff
- **User-Agent**: Required — set a descriptive string: `MyTool/1.0 (mailto:[email protected])`
- Persistent abuse may result in IP-level blocks

## Academic Use Cases

- **Formula extraction for ML training** — Build equation datasets with ground-truth LaTeX notation, free of OCR noise from PDF parsing
- *

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.