Academic Web ScrapingCAUTION

🔬 A curated collection of 23,000+ agent skills for empirical research across 8 social science disciplines. | 精选 23,000+ AI Agent 技能库,覆盖8大社会科学学科的实证研究。CoPaper.AI 20分钟完成一篇可复现的规范实证论文,并支持用户上传 Skills。-- Maintained by CoPaper.AI from Stanford REAP.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: academic-web-scraping
description: "Ethical web scraping and API-based data collection for research"
metadata:
  openclaw:
    emoji: "🌐"
    category: "tools"
    subcategory: "scraping"
    keywords: ["web scraping", "API data collection", "web search strategies", "data extraction"]
    source: "N/A"
---

# Academic Web Scraping Guide

## Overview

Research often requires collecting data from the web -- whether it is bibliographic metadata from academic databases, experimental datasets from public repositories, social media posts for computational social science, or economic indicators from government portals. Web scraping and API-based data collection are essential skills for modern researchers across disciplines.

This guide covers both approaches: structured API access for platforms that provide one, and web scraping for when no API exists. It emphasizes ethical data collection practices, including respecting robots.txt, rate limiting, terms of service compliance, and IRB considerations for human-subject data. The goal is to collect research data reliably and responsibly.

Whether you are building a dataset for a machine learning paper, collecting metadata for a systematic review, or gathering public data for policy research, these patterns help you do it correctly and efficiently.

## API-Based Data Collection

APIs are always preferable to scraping when available. They provide structured data, are officially supported, and have clear usage terms.

### Academic APIs

| API | Data | Rate Limit | Auth |
|-----|------|-----------|------|
| OpenAlex | Papers, authors, venues, concepts | 100K req/day | Email in header |
| Crossref | DOI metadata | 50 req/sec (polite pool) | Email in header |
| PubMed (Entrez) | Biomedical literature | 10 req/sec (with key) | API key (free) |
| arXiv | Preprints | 1 req/3sec | None |
| CORE | Open access papers | 10 req/sec | API key (free) |

### Example: Collecting Papers from OpenAlex

```python
import requests
import time

class OpenAlexClient:
    BASE_URL = "https://api.openalex.org"

    def __init__(self, email):
        self.session = requests.Session()
        self.session.headers.update({
            'User-Agent': f'ResearchBot/1.0 (mailto:{email})'
        })

    def search_works(self, query, filters=None, per_page=25, max_results=100):
        """Search for works with optional filters."""
        results = []
        page = 1

        while len(results) < max_results:
            params = {
                'search': query,
                'per_page': min(per_page, max_results - len(results)),
                'page': page,
            }
            if filters:
                params['filter'] = ','.join(f'{k}:{v}' for k, v in filters.items())

            resp = self.session.get(f'{self.BASE_URL}/works', params=params)
            resp.raise_for_status()
            data = resp.json()

            works = data.get('results', [])
            if not works:
                break

            results.extend(works)
            page += 1
            time.sleep(0.1)  # Polite rate limiting

        return results[:max_results]

    def get_work(self, openalex_id):
        """Get a single work by OpenAlex ID."""
        resp = self.session.get(f'{self.BASE_URL}/works/{openalex_id}')
        resp.raise_for_status()
        return resp.json()

# Usage
client = OpenAlexClient(email="[email protected]")
papers = client.search_works(
    "transformer attention mechanism",
    filters={
        'publication_year': '2023-2024',
        'type': 'journal-article',
        'open_access.is_oa': 'true'
    },
    max_results=200
)

for paper in papers[:5]:
    print(f"- {paper['title']} ({paper['publication_year']})")
    print(f"  DOI: {paper['doi']}")
    print(f"  Citations: {paper['cited_by_count']}")
```

### Example: PubMed Entrez API

```python
from Bio import Entrez

Entrez.email = "[email protected]"
Entrez.api_key = os.environ.get("NCBI_API_KEY")  # optional

def search_pubmed(query, max_results=100):
    """Search PubMed and retrieve article details."""
    # Search
    handle = Entrez.esearch(db="pubmed", term=query,
                            retmax=max_results, sort="relevance")
    search_results = Entrez.read(handle)
    id_list = search_results["IdList"]

    if not id_list:
        return []

    # Fetch details
    handle = Entrez.efetch(db="pubmed", id=id_list,
                           rettype="xml", retmode="xml")
    records = Entrez.read(handle)

    articles = []
    for article in records['PubmedArticle']:
        medline = article['MedlineCitation']
        art_info = medline['Article']
        articles.append({
            'pmid': str(medline['PMID']),
            'title': art_info.get('ArticleTitle', ''),
            'abstract': art_info.get('Abstract', {}).get(
                'AbstractText', [''])[0] if 'Abstract' in art_info else '',
            'journal': art_info['Journal']['Title'],
            'year': art_info['Journal']['JournalIssue'].get(
                'PubDate', {}).get('Year', ''),
        })

    return articles
```

## Web Scraping Fundamentals

When no API exists, scraping becomes necessary. Always check for an API first.

### Tools Comparison

| Tool | Type | JavaScript Support | Speed | Learning Curve |
|------|------|-------------------|-------|---------------|
| requests + BeautifulSoup | HTTP + parsing | No | Fast | Low |
| Scrapy | Framework | No (without middleware) | Very fast | Medium |
| Selenium | Browser automation | Yes | Slow | Medium |
| Playwright | Browser automation | Yes | Medium | Medium |
| httpx | Async HTTP | No | Very fast | Low |

### Basic Scraping with BeautifulSoup

```python
import requests
from bs4 import BeautifulSoup
import time

def scrape_conference_proceedings(url, delay=2.0):
    """Scrape paper titles and links from a conference page."""
    headers = {
        'User-Agent': 'ResearchBot/1.0 (Academic research; [email protected])'
    }

    respo

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (1)

| CORE | Open access papers | 10 req/sec | API key (free) |

Gates applied: no_behavioural_pass.