Stanford University Scraper

An Apify actor that crawls Stanford Profiles directory pages and extracts detailed faculty information in a single, automated workflow. It combines listing and detail scraping to output a unified dataset, eliminating the need to run separate actors.

Overview

The actor performs a two-phase scrape:
1. List Phase: Starts from a Stanford directory page (default is School of Humanities & Sciences), follows pagination links, and collects all valid profile URLs.
2. Detail Phase: Fetches each profile page to extract structured data including contact info, academic roles, and professional history.

It includes smart logic to stop crawling when no new links are found, preventing infinite loops and wasted resources.

Key Features

• Unified Workflow: Handles both discovery and data extraction in one actor, producing a single, clean dataset.
• Robust Pagination: Uses <link rel="next"> tags and URL patterns (?p=) for reliable navigation, with automatic halt on duplicate pages.
• Parallel Execution: Configurable concurrency for detail scraping to speed up data collection.
• Flexible Input Sources: Start from a directory URL, add specific profile URLs via text, or pull URLs from an existing Apify dataset.

How to Use

Configure the actor using the input fields below and run it. It will crawl the directory and profile pages, outputting all results into one dataset.

Input

Output

The actor outputs a dataset of JSON objects, one per profile. Each item includes:

{
  "url": "https://profiles.stanford.edu/jane-doe",
  "name": "Jane Doe",
  "email": "jane.doe@stanford.edu",
  "departments": ["History"],
  "faculty": ["School of Humanities and Sciences"],
  "personalWebsite": "https://janedoe.com",
  "bio": "Professor of History…",
  "academicAppointments": ["Associate Professor (2020–)"],
  "professionalEducation": ["PhD, Stanford University (2015)"],
  "honorsAwards": ["Guggenheim Fellowship (2023)"]
}

Use Cases: Academic research, faculty lead generation, or populating internal databases/CRMs with structured Stanford data.

For issues or feedback, please use the actor's issue tracker.