Substack Newsletter Scraper

Overview

An Apify Actor that scrapes Substack newsletters for publication details, author contact info, and post data. Use it for lead generation, market research, or competitive analysis by processing multiple newsletters in a single run.

Key Features

• Publication Metadata: Extracts title, description, author info, categories, and public subscriber/follower counts.
• Contact Discovery: Finds author email addresses and scrapes social media links (Twitter, LinkedIn, etc.).
• Flexible Post Scraping: Choose from three modes to control data depth and usage:
  • none: Newsletter metadata only (fastest).
  • information: Metadata plus post titles, dates, and engagement stats.
  • information_and_content: All of the above plus full article content.
• Batch Processing: Scrape multiple newsletter URLs in one execution.
• Pay-per-Event: You are only charged for successfully scraped newsletters.

How to Use

Configure the actor with input parameters via JSON. The only required parameter is the list of newsletter URLs.

Input

Provide input as a JSON object. The actor supports both *.substack.com URLs and custom domains.

Required:
* newsletterUrls (array): A list of newsletter objects, each containing a url.

Optional:
* postScrapingMode (string): Default is "none". Options: "none", "information", "information_and_content".
* maxPostsPerNewsletter (number): Default is 12 (max: 12). Number of recent posts to scrape. Only applies if postScrapingMode is not "none".
* delayBetweenRequests (number): Default is 3000 ms (range: 500-10000). Delay between requests to avoid rate limiting.

Example Input

{
  "newsletterUrls": [
    { "url": "https://platformer.substack.com" },
    { "url": "https://lennysnewsletter.com" }
  ],
  "postScrapingMode": "information",
  "maxPostsPerNewsletter": 5,
  "delayBetweenRequests": 2000
}

Output

The output structure depends on the postScrapingMode.

All modes include comprehensive newsletter metadata:
* Author details (ID, name, bio, photo, handle).
* Contact info (email, website, social media URLs).
* Publication details (ID, name, description, logo, creation date).
* Counts (subscribers, followers, visibility flags).
* Status flags (is paid, is active, etc.).

If postScrapingMode is "information" or "information_and_content", the output adds a posts array. Each post object contains:
* url, title, subtitle, published_at.
* Engagement metrics (like_count, comment_count).
* word_count and a preview_text.
* When mode is "information_and_content", it also includes the full article_content_html.

Example Output Snippet (Mode: "information")

{
  "author_name": "Casey Newton",
  "email": "casey@platformer.news",
  "publication_name": "Platformer",
  "subscriber_count": 176000,
  "posts": [
    {
      "title": "Trump won. Here's what comes next.",
      "published_at": "2024-11-06T14:30:00.000Z",
      "like_count": 245,
      "comment_count": 89,
      "word_count": 1250
    }
  ]
}