Twitter Profile Scraper

An Apify Actor for scraping data from Twitter/X profiles. It extracts comprehensive profile details and tweets, including engagement metrics, media, and metadata. Authentication is required to access recent chronological tweets.

Overview

This actor scrapes public Twitter profiles. Without providing authentication cookies, the results will be limited to older "Highlight" tweets. With valid cookies, it scrapes the profile's recent chronological timeline.

Key Features

• Profile Data: Extracts username, display name, bio, location, website, join date, verification status, follower/following counts, and profile/banner image URLs.
• Complete Tweet Data: Captures full text, ID, URL, timestamp, and type (regular, reply, retweet).
• Engagement Metrics: Gets likes, retweets, replies, quotes, bookmarks, views, and total engagement counts.
• Media & Context: Extracts media URLs, external links, hashtags, mentions, and reply/retweet context.
• Authentication Support: Uses provided cookies to access recent tweets and private/protected profiles.
• Stealth & Reliability: Includes human-like behavior simulation (random delays, scrolling), rate limit protection, and block detection.
• Filtering: Options to include or exclude replies and retweets.
• Multi-Profile: Scrape multiple usernames in a single run.

How to Use

Authentication Requirement

To scrape recent tweets, you must provide Twitter authentication cookies in the cookies input field. Without cookies, only older "Highlight" tweets are available.

Input Parameters

Required:
* usernames (array): List of Twitter handles to scrape, without the "@". Example: ["elonmusk", "openai"]

Key Optional Parameters:
* maxTweets (integer): Max tweets per profile (1-500). Default: 50.
* includeReplies (boolean): Include reply tweets. Default: false.
* includeRetweets (boolean): Include retweets. Default: true.
* cookies (string): Twitter authentication cookies in JSON format.
* sessionName (string): Name for persisting cookies between runs. Default: "default_session".
* minDelayBetweenRequests / maxDelayBetweenRequests (integer): Range for random delays between actions (in seconds) to mimic human behavior.
* humanizeBehavior (boolean): Enable human-like interactions. Default: true.

Input/Output

Input Example

{
  "usernames": ["github"],
  "maxTweets": 100,
  "includeReplies": false,
  "cookies": "[{\"name\":\"auth_token\",\"value\":\"abc123...\"}]"
}

Output Format

The actor outputs a JSON object for each profile. The structure includes profile metadata and an array of posts.

{
  "username": "github",
  "display_name": "GitHub",
  "bio": "The world’s leading AI-powered developer platform.",
  "location": "San Francisco, CA",
  "website": "https://github.com",
  "joined_date": "Joined January 2008",
  "followers_count": 3500000,
  "following_count": 152,
  "verified": true,
  "profile_image_url": "https://pbs.twimg.com/...",
  "scraped_at": "2024-01-15T12:00:00.000Z",
  "posts": [
    {
      "tweet_id": "1234567890",
      "tweet_url": "https://twitter.com/github/status/1234567890",
      "text": "Tweet content here",
      "author_username": "github",
      "timestamp": "2024-01-15T10:30:00.000Z",
      "likes_count": 15000,
      "retweets_count": 3000,
      "replies_count": 450,
      "views_count": 250000,
      "media_urls": ["https://pbs.twimg.com/media/..."]
    }
  ]
}