Google Scholar Search Scraper

This actor automates data extraction from Google Scholar search results. It's designed for researchers, data scientists, and developers who need to programmatically collect academic literature, citations, and publication details at scale.

Overview

The scraper navigates Google Scholar, handles pagination, and extracts structured data from search result pages. You provide search URLs, and it returns clean, organized data, eliminating the need for manual copying and pasting. It's useful for literature reviews, bibliometric analysis, and building academic datasets.

Key Features

• Structured Data Extraction: Pulls titles, authors, publication years, citations, summaries, and links.
• Handles Complex Queries: Accepts direct Google Scholar URLs with filters (year, language, etc.).
• Automatic Pagination: Can navigate through multiple pages of results.
• Proxy Support: Uses Apify Proxy (including residential proxies) to help avoid bot detection and manage request rates.
• Configurable Limits: Set maximum items per search and retry attempts for failed requests.

How to Use

Configure the actor using a JSON input. The core requirement is a list of urls (Google Scholar search result pages). Run the actor on the Apify platform, and it will output the scraped data into a dataset.

Input/Output

Input Format

Configure the actor with a JSON object. Here's the structure with explanations:

{
  "max_retries_per_url": 2,
  "proxy": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "SG"
  },
  "max_items_per_url": 20,
  "urls": [
    "https://scholar.google.com/scholar?hl=vi&as_sdt=0%2C5&as_ylo=2025&q=AI&btnG=",
    "https://scholar.google.com/scholar?start=80&q=webscraping&hl=vi&as_sdt=0,5&as_ylo=2025"
  ]
}

• urls (Required): An array of Google Scholar search URLs. You can generate these by performing a search in your browser and copying the address.
• max_items_per_url: Limits the total number of items scraped per URL.
• max_retries_per_url: Number of retry attempts if a request fails.
• proxy: Configuration for Apify Proxy. Using residential proxies (RESIDENTIAL) is recommended to reduce the chance of being blocked.

Output Format

The actor stores results in a dataset. Each item typically includes fields like:
* title
* authors
* year
* citationCount
* summary (snippet)
* publication (source/journal)
* url (link to the paper)
* scholarUrl (link to the Google Scholar entry)

Contact

If you encounter issues, contact the developer via My profile.