Newsletter Scraper

Extracts complete archives from Substack, Beehiiv, and Ghost newsletters. Outputs clean, structured data including post content, metadata, and engagement metrics, formatted for AI training or content analysis.

Overview

This actor scrapes newsletter platforms to provide full post histories. It outputs content in multiple formats (Markdown, HTML, plain text) alongside calculated metrics like word and token counts. It's designed for use cases like building LLM training datasets, content research, or competitive analysis.

Key Features

• Platform Support: Works with Substack, Beehiiv, and Ghost newsletters.
• Complete Data Extraction:
  • Full post content (title, body, author, publication date).
  • Newsletter metadata (name, description, author info).
  • Engagement metrics (likes, comments) where available.
  • Image data (URLs, alt text, dimensions).
  • Paywall/premium content detection.
• LLM-Optimized Output: Provides clean Markdown and estimates token counts (GPT-style).
• Flexible Scraping Modes: Choose to scrape the full archive, a single post, or just newsletter info.
• Rate-Limiting: Configurable delay between requests to respect target sites.

How to Use

Input Configuration

1. Newsletter URL: Provide the homepage URL of the target newsletter (e.g., https://newsletter.substack.com).
2. Scrape Mode:
   • FULL_ARCHIVE: Gets the entire post history.
   • SINGLE_POST: Extracts one specific post URL.
   • NEWSLETTER_INFO: Fetches only metadata (name, description, author).
3. Output Format: Choose MARKDOWN (recommended for AI), HTML, PLAIN_TEXT, or ALL.
4. Optional Settings:
   • maxPosts: Limit the number of posts scraped (0 for no limit).
   • includeImages: Toggle image data extraction.
   • includeMetadata: Toggle engagement metrics.
   • postsSinceDate: Scrape only posts published after this date.
   • delayBetweenRequests: Set a pause (in seconds) between page requests.

Execution

Run the actor with your configured input. It will:
1. Validate the provided URL.
2. Extract post links from the archive.
3. Scrape content and metadata from each post.
4. Process data into the chosen output format(s).
5. Save the results to the dataset.

Input/Output

Input Example (JSON):

{
  "newsletterUrl": "https://example.beehiiv.com",
  "scrapeMode": "FULL_ARCHIVE",
  "outputFormat": "MARKDOWN",
  "maxPosts": 50,
  "includeImages": true
}

Output Structure:
The dataset contains an array of post objects. Each object includes:

Results are accessible via the Apify platform's Dataset tab in clean JSON format.