Google Scholar ScraperSAFE

🔬 A curated collection of 23,000+ agent skills for empirical research across 8 social science disciplines. | 精选 23,000+ AI Agent 技能库,覆盖8大社会科学学科的实证研究。CoPaper.AI 20分钟完成一篇可复现的规范实证论文,并支持用户上传 Skills。-- Maintained by CoPaper.AI from Stanford REAP.

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: google-scholar-scraper
description: "Ethical Google Scholar data collection techniques and best practices"
metadata:
  openclaw:
    emoji: "🔍"
    category: "tools"
    subcategory: "scraping"
    keywords: ["Google Scholar", "web scraping", "bibliometrics", "citation data", "scholarly search", "rate limiting"]
    source: "wentor-research-plugins"
---

# Google Scholar Scraper

A skill for ethically collecting bibliometric data from Google Scholar, including search results, citation counts, author profiles, and related articles. Covers rate limiting, CAPTCHA avoidance, alternative APIs, legal considerations, data parsing, and practical workflows that balance data needs with responsible access.

## Legal and Ethical Considerations

### Before You Scrape

Google Scholar does not offer an official API, and its Terms of Service restrict automated access. Researchers must weigh their data needs against legal and ethical constraints.

```
Legal landscape:

Terms of Service:
  - Google's ToS prohibit automated queries
  - Violation can result in IP blocking (temporary or permanent)
  - Institutional IPs can be blocked, affecting all campus users
  - In some jurisdictions, ToS violations are not legally binding
    for non-commercial academic research, but this is debated

Ethical guidelines:
  - Minimize load: respect the server, use delays between requests
  - Cache aggressively: never request the same page twice
  - Use official alternatives first (see below)
  - Do not redistribute raw scraped data
  - Cite Google Scholar as your data source in publications
  - Consider whether your research question truly requires
    Google Scholar data, or if Web of Science, Scopus, or
    OpenAlex could answer it instead

Official and semi-official alternatives:
  - OpenAlex API: free, no key required, excellent coverage
  - OpenAlex API: free, comprehensive, well-documented
  - Crossref API: free, DOI-based metadata and citation counts
  - CORE API: free, full-text open access content
  - Google Scholar Alerts: manual but ToS-compliant monitoring
  - Publish or Perish (software): uses Google Scholar with built-in
    rate limiting, commonly used in bibliometric research
```

## Data Collection Approaches

### Using Scholarly (Python Library)

The `scholarly` Python library wraps Google Scholar access with built-in rate limiting and proxy support. It is the most commonly used tool for academic Google Scholar scraping.

```python
from scholarly import scholarly, ProxyGenerator

def setup_scholarly_with_proxy():
    """
    Configure scholarly with a free proxy to reduce blocking risk.
    For heavy usage, consider ScraperAPI or similar paid services.
    """
    pg = ProxyGenerator()
    # Free proxy (less reliable, suitable for small jobs)
    pg.FreeProxies()
    scholarly.use_proxy(pg)


def search_scholar(query, max_results=20):
    """
    Search Google Scholar and collect structured results.

    IMPORTANT: Add delays between queries to avoid blocking.
    Recommended: 10-30 seconds between searches.
    """
    import time

    results = []
    search_query = scholarly.search_pubs(query)

    for i in range(max_results):
        try:
            result = next(search_query)
            parsed = {
                "title": result["bib"].get("title", ""),
                "author": result["bib"].get("author", []),
                "year": result["bib"].get("pub_year", ""),
                "venue": result["bib"].get("venue", ""),
                "abstract": result["bib"].get("abstract", ""),
                "citations": result.get("num_citations", 0),
                "url": result.get("pub_url", ""),
            }
            results.append(parsed)

            # Rate limiting: wait between result fetches
            time.sleep(2)

        except StopIteration:
            break

    return results


def get_author_profile(author_name):
    """
    Retrieve an author's Google Scholar profile.
    Includes h-index, i10-index, and publication list.
    """
    search_query = scholarly.search_author(author_name)
    author = next(search_query)
    author = scholarly.fill(author)

    profile = {
        "name": author.get("name", ""),
        "affiliation": author.get("affiliation", ""),
        "h_index": author.get("hindex", 0),
        "i10_index": author.get("i10index", 0),
        "cited_by": author.get("citedby", 0),
        "interests": author.get("interests", []),
        "publications": len(author.get("publications", [])),
    }

    return profile
```

## Rate Limiting and Anti-Blocking

### Best Practices

```
Rate limiting strategy:

1. Request delays:
   - Between search queries: 15-30 seconds minimum
   - Between profile lookups: 10-20 seconds
   - Between citation fetches: 5-10 seconds
   - Add random jitter: delay + random(0, 5) seconds

2. Session management:
   - Rotate user agents (maintain a list of 10+ real browser UAs)
   - Clear cookies periodically
   - Use residential proxies for large jobs (paid)
   - Limit sessions to 100-200 requests before rotating proxy

3. Caching:
   - Cache every response to disk (shelve, sqlite, or JSON)
   - Check cache before making any request
   - Set cache expiry (7-30 days for citation counts)

4. Batch scheduling:
   - Spread collection over days, not hours
   - Run during off-peak hours (late night UTC)
   - Process in batches of 50-100 queries per session
```

### Handling CAPTCHAs and Blocks

```python
import time
import random

def resilient_search(query, max_retries=3):
    """
    Search with exponential backoff on failures.
    When blocked, wait and retry with increasing delays.
    """
    for attempt in range(max_retries):
        try:
            results = search_scholar(query, max_results=10)
            return results
        except Exception as e:
            if "CAPTCHA" in str(e) or "429" in str(e):
                wait_time = (2 ** attempt) * 60 + random.randint(0, 30)
                print(f"Blocked. Waiting {wait_time}s 

SAFEgrade B · trust 89/100 Nothing in the source contradicts what it says it does. Grade A is reserved for packages that have also passed the behavioural sandbox.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (0)

No findings outside the package's declared scope.

Gates applied: no_behavioural_pass.