NextdoorScraper

A web scraper built with Playwright and Crawlee that extracts recommendation posts and their replies from Nextdoor pages. Outputs structured JSON data for automation and analysis.

Key Features

• Extracts recommendation posts with author details, dates, and content.
• Captures nested replies within recommendation threads.
• Collects reaction counts and types (likes, comments, shares).
• Handles dynamic content, cookie banners, and address selection prompts.
• Includes robust error handling, retry logic, and configurable timeouts.
• Outputs clean, structured JSON.

How to Use

Setup & Run

1. Install dependencies with npm install.
2. Configure the input in storage/key_value_stores/default/INPUT.json:
   json { "startUrls": ["https://nextdoor.com/pages/your-target-page/"], "maxRequestsPerCrawl": 50, "cutOffDate": "2024-01-01T00:00:00.000Z" }
3. Execute the scraper with npm start.

Note: Some pages may require address selection or authentication, which the scraper attempts to handle. Access restrictions for certain pages may still occur.

Input Parameters

• startUrls (required): List of Nextdoor page URLs to scrape.
• maxRequestsPerCrawl: Limits the number of pages scraped per run.
• cutOffDate (optional): An ISO date string. Only posts newer than this date will be scraped. Omit to scrape all posts.

Input & Output

Input Format

Configure the scraper via the INPUT.json file as shown in the How to Use section.

Output Structure

Scraped data is saved in storage/datasets/default/ as JSON. Each result includes the page URL, title, summary, an array of posts, and a timestamp.

{
  "url": "https://nextdoor.com/pages/example/",
  "title": "Page Title",
  "summary": "Page summary if available",
  "posts": [
    {
      "author": { "name": "S. H.", "initials": "S", "avatar": "https://example.com/avatar.jpg" },
      "date": "20 Nov",
      "content": "Post content text",
      "reactions": { "total": 16, "types": ["like", "comment", "share"] },
      "replies": [
        {
          "author": { "name": "D. W.", "initials": "D" },
          "date": "20 Nov",
          "content": "Reply content",
          "reactions": { "total": 1, "types": ["like"] }
        }
      ]
    }
  ],
  "scraped_at": "2024-01-15T10:30:00.000Z"
}

Troubleshooting

• Address Selection Required: The scraper will log a message if it detects this prompt.
• Rate Limiting: Try reducing maxRequestsPerCrawl or adding delays between requests.
• Missing Data: The scraper uses fallback values to maintain consistent output structure.

Technical Stack

• Playwright for browser automation.
• Crawlee for crawling infrastructure.
• TypeScript for type safety.
• Apify SDK for data management.