Bluesky Jetstream Scraper

This Apify actor collects real-time data from the Bluesky social network via its ATProto Firehose (Jetstream). It filters the live stream of content by specific criteria and outputs structured data.

Important context:
* Jetstream vs. Crawling: It uses Bluesky's efficient firehose API for a continuous, real-time data stream. This avoids the rate limits and overhead of traditional crawling.
* Real-Time Only: It captures data as it happens. It cannot retrieve historical posts.
* Evolving Platform: Bluesky's APIs are still in development; the actor may require updates to maintain compatibility.

Key Features

• Real-time streaming from the Bluesky firehose.
• Filterable output by hashtags, users, languages, and event types.
• Configurable data formats (JSON, CSV, etc.).
• Efficient operation using the native Jetstream API.

How to Use

Configure the actor's input parameters in Apify to define your data collection. The scraper will run continuously, outputting matching posts and events to the dataset until stopped.

Input Parameters

Filtering Parameters

• hashtags (Array of strings): Collect posts containing any of these hashtags (omit the #). Example: ["apify", "scraping"].
• usernames (Array of strings): Collect posts authored by any of these users. Example: ["user1.bsky.social", "user2.bsky.social"].
• languages (Array of strings): Collect posts detected in any of these language codes. Example: ["en", "pt"].
• wantedCollections (Array of strings): Define which Bluesky event types to collect. Common options:
  • app.bsky.feed.post: Standard posts.
  • app.bsky.feed.like: Likes.
  • app.bsky.feed.repost: Reposts.
  • app.bsky.graph.follow: Follows.

Processing & Output Parameters

• detectLanguage (Boolean): Enables automatic language detection for posts without a language tag.
• outputFormat (String): Chooses the dataset file format (e.g., json, csv).
• customOutputFields (Array of strings): Specifies which post fields to include in the output, allowing you to limit data to only what you need.

Output

The actor outputs items to the Apify dataset, with fields varying based on the event type (wantedCollections) and your customOutputFields. For a standard app.bsky.feed.post, typical output includes:

{
  "uri": "at://did:plc:abc123/app.bsky.feed.post/3k44xq",
  "text": "Post content here #example",
  "author": {
    "did": "did:plc:def456",
    "handle": "user.bsky.social"
  },
  "indexedAt": "2024-01-01T12:00:00.000Z",
  "langs": ["en"],
  "hashtags": ["example"]
}

You can access results via the Apify API, or download them directly in your chosen format (JSON, CSV, etc.) from the Apify Console at https://console.apify.com.