Twitter Hashtag Tweet Scraper

Scrapes tweets based on hashtags using an authenticated Twitter/X session. It handles rate limits automatically and outputs structured tweet data to an Apify dataset.

Key Features

• Multi-hashtag scraping: Collect tweets from multiple hashtags in one run.
• Rate limit handling: Automatically pauses when Twitter/X limits are hit and uses smart delays between requests.
• Deduplication: Avoids duplicate tweets using unique tweet IDs.
• Comprehensive data: Extracts full tweet text, author info, engagement metrics (likes, retweets, replies), and metadata.
• Time filtering: Limits tweets by age using the max_age_hint_minutes parameter.
• Real-time streaming: Pushes data to the dataset in batches as it's collected.

How to Use

Required Input

You must provide two parameters:
1. hashtags: An array of hashtags to search (e.g., ["AI", "MachineLearning"]). Do not include the # symbol.
2. cookies: An array of Twitter/X session cookies for authentication. You must be logged in.

Getting Your Cookies

1. Log into Twitter/X in your browser.
2. Open Developer Tools (F12).
3. Go to the Application or Storage tab and find Cookies for x.com or twitter.com.
4. Copy the auth_token and ct0 cookies. The twid cookie is optional but recommended.
5. Format them for the actor input:
   json "cookies": [ { "name": "auth_token", "value": "YOUR_TOKEN", "domain": ".x.com" }, { "name": "ct0", "value": "YOUR_CT0_TOKEN", "domain": ".x.com" } ]

Input Example

{
  "hashtags": ["AI", "DataScience"],
  "max_tweets": 3000,
  "max_age_hint_minutes": 720,
  "cookies": [
    { "name": "auth_token", "value": "your_auth_token", "domain": ".x.com" },
    { "name": "ct0", "value": "your_ct0_token", "domain": ".x.com" }
  ]
}

Optional Parameters

• max_tweets: (Default: 5000) The maximum total number of tweets to collect across all hashtags.
• max_age_hint_minutes: (Default: 1440 / 24 hours) The maximum age of tweets to collect, in minutes.

Output

The actor stores tweets as individual items in the Apify dataset. Each item contains this structure:

{
  "id": "1234567890123456789",
  "text": "This is a sample tweet with #hashtag",
  "author": "John Doe",
  "username": "johndoe",
  "created_at": "2024-01-15T10:30:00Z",
  "retweet_count": 42,
  "like_count": 156,
  "reply_count": 23,
  "quote_count": 8,
  "url": "https://twitter.com/johndoe/status/1234567890123456789",
  "hashtags": ["hashtag"],
  "mentions": ["mention1"],
  "is_retweet": false,
  "language": "en",
  "user_followers": 1500,
  "user_verified": false,
  "search_hashtag": "AI",
  "scraped_at": "2024-01-15T11:00:00Z"
}

Key Output Fields

• id: Unique tweet ID.
• text: Full content of the tweet.
• author/username: Display name and Twitter handle.
• created_at: When the tweet was posted (ISO timestamp).
• retweet_count/like_count/reply_count/quote_count: Engagement metrics.
• url: Direct link to the tweet.
• hashtags/mentions: Arrays extracted from the tweet text.
• user_followers/user_verified: Author profile info.
• search_hashtag: Which input hashtag found this tweet.
• scraped_at: When the actor collected this data.

Performance Notes

• The actor distributes the max_tweets limit evenly across the provided hashtags.
• It processes data in batches for efficiency and streams it to the dataset continuously.
• All rate limiting is handled automatically; the actor will pause and resume as needed.