Grobid Pdf ParsingSAFE

🔬 A curated collection of 23,000+ agent skills for empirical research across 8 social science disciplines. | 精选 23,000+ AI Agent 技能库,覆盖8大社会科学学科的实证研究。CoPaper.AI 20分钟完成一篇可复现的规范实证论文,并支持用户上传 Skills。-- Maintained by CoPaper.AI from Stanford REAP.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: grobid-pdf-parsing
description: "Extract structured text, metadata, and references from academic PDFs"
metadata:
  openclaw:
    emoji: "📄"
    category: "tools"
    subcategory: "document"
    keywords: ["PDF parsing", "PDF extraction", "document chunking", "format conversion"]
    source: "https://github.com/kermitt2/grobid"
---

# GROBID PDF Parsing Guide

## Overview

Academic PDFs are the primary format for distributing research, yet extracting structured data from them remains challenging. PDFs encode visual layout, not semantic structure -- headings, paragraphs, equations, tables, and citations are all just positioned text and graphics. GROBID (GeneRation Of BIbliographic Data) is the leading open-source tool for parsing academic PDFs into structured XML/TEI format, extracting metadata, body text, references, and figures with high accuracy.

GROBID is used by major academic platforms including CORE, ResearchGate, and others for large-scale document processing. It combines machine learning models (CRF and deep learning) with heuristic rules to handle the diverse formatting of academic papers across publishers and disciplines.

This guide covers installing and running GROBID, using its REST API for batch processing, extracting specific elements (metadata, references, body sections), and integrating GROBID output into downstream workflows such as knowledge bases, systematic reviews, and literature analysis pipelines.

## Installation

### Docker (Recommended)

```bash
# Pull the latest GROBID image
docker pull grobid/grobid:0.8.1

# Run GROBID server
docker run --rm --init \
  --ulimit core=0 \
  -p 8070:8070 \
  grobid/grobid:0.8.1

# GROBID is now running at http://localhost:8070
# Web console: http://localhost:8070/console
```

### From Source

```bash
git clone https://github.com/kermitt2/grobid.git
cd grobid
./gradlew clean install
./gradlew run
```

## REST API Usage

### Process Full Document

```bash
# Process a single PDF and get TEI XML
curl -v --form [email protected] \
  http://localhost:8070/api/processFulltextDocument \
  -o paper.tei.xml

# With options
curl -v --form [email protected] \
  --form consolidateHeader=1 \
  --form consolidateCitations=1 \
  --form includeRawCitations=1 \
  http://localhost:8070/api/processFulltextDocument \
  -o paper.tei.xml
```

### API Endpoints

| Endpoint | Purpose | Input | Output |
|----------|---------|-------|--------|
| `/api/processFulltextDocument` | Full paper parsing | PDF | TEI XML |
| `/api/processHeaderDocument` | Metadata only | PDF | TEI XML (header) |
| `/api/processReferences` | Reference parsing | PDF | TEI XML (refs) |
| `/api/processCitation` | Parse citation string | Text | TEI XML |
| `/api/processDate` | Parse date string | Text | Structured date |

### Python Client

```python
import requests
from pathlib import Path

class GrobidClient:
    def __init__(self, base_url='http://localhost:8070'):
        self.base_url = base_url

    def process_fulltext(self, pdf_path, consolidate_header=True,
                         consolidate_citations=True):
        """Process a PDF and return TEI XML."""
        url = f'{self.base_url}/api/processFulltextDocument'
        files = {'input': open(pdf_path, 'rb')}
        data = {
            'consolidateHeader': '1' if consolidate_header else '0',
            'consolidateCitations': '1' if consolidate_citations else '0',
        }
        response = requests.post(url, files=files, data=data)
        response.raise_for_status()
        return response.text

    def process_header(self, pdf_path):
        """Extract only header metadata from PDF."""
        url = f'{self.base_url}/api/processHeaderDocument'
        files = {'input': open(pdf_path, 'rb')}
        response = requests.post(url, files=files)
        response.raise_for_status()
        return response.text

    def is_alive(self):
        """Check if GROBID server is running."""
        try:
            resp = requests.get(f'{self.base_url}/api/isalive')
            return resp.status_code == 200
        except requests.ConnectionError:
            return False

# Usage
client = GrobidClient()
if client.is_alive():
    tei_xml = client.process_fulltext('paper.pdf')
    with open('paper.tei.xml', 'w') as f:
        f.write(tei_xml)
```

## Parsing TEI XML Output

### Extracting Metadata

```python
from lxml import etree

def parse_tei_metadata(tei_xml):
    """Extract title, authors, abstract from TEI XML."""
    ns = {'tei': 'http://www.tei-c.org/ns/1.0'}
    root = etree.fromstring(tei_xml.encode('utf-8'))

    # Title
    title_el = root.find('.//tei:titleStmt/tei:title', ns)
    title = title_el.text if title_el is not None else ''

    # Authors
    authors = []
    for author in root.findall('.//tei:sourceDesc//tei:author', ns):
        forename = author.findtext('.//tei:forename', '', ns)
        surname = author.findtext('.//tei:surname', '', ns)
        if surname:
            authors.append(f'{forename} {surname}'.strip())

    # Abstract
    abstract_el = root.find('.//tei:profileDesc/tei:abstract', ns)
    abstract = ''.join(abstract_el.itertext()).strip() if abstract_el is not None else ''

    # DOI
    doi_el = root.find('.//tei:idno[@type="DOI"]', ns)
    doi = doi_el.text if doi_el is not None else ''

    return {
        'title': title,
        'authors': authors,
        'abstract': abstract,
        'doi': doi,
    }
```

### Extracting Body Sections

```python
def parse_tei_sections(tei_xml):
    """Extract structured sections from TEI XML body."""
    ns = {'tei': 'http://www.tei-c.org/ns/1.0'}
    root = etree.fromstring(tei_xml.encode('utf-8'))

    sections = []
    for div in root.findall('.//tei:body/tei:div', ns):
        head = div.findtext('tei:head', '', ns).strip()
        paragraphs = []
        for p in div.findall('tei:p', ns):
            text = ''.join(p.itertext()).strip()
            if text:
                paragraphs.append(text)
        sections.append({
            

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.