Discord Message Scraper

🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🚜 🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽🌽 # Harvest Edge / Discord Message Scraper 🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱 🌽🌽🌽🌽🍆🌽🌽🌽🌽🚜🌱🌱🌱🌱🌱🌱 Harvest Edge makes business information available to everyone! ## Overview Easily scrape Discord messages with the Discord Message Scraper, a powerful Apify actor designed for developers and data analysts. This actor extracts messages, attachments, embeds, reactions, and mentions from any Discord channel or guild, using the Discord API. With support for scraping in both “newer” and “older” directions and a customizable 60-second timeout, it ensures reliable data collection for analytics, research, or archiving. Feel free to contact us via Apify for feature requests or bug reports. ## How to Get Your Token - Open developer tools on your browser, by pressing F12, and go to discord.com/channels/@me - Click the 'Network' tab in the developer tools, and find the file named @me. You can filter for Fetch/XHR requests to help find it. Refresh the page if you do not see it. - Be sure you are looking at the 'Headers' of this file and scroll down to 'Request Headers' - Your token is the code that is listed as 'Authorization' ## Features - Fetches up to 10,000 messages per run from a specified Discord channel. - Supports scraping in batches, with the option of going either direction from a starting message. - Outputs detailed message data (content, author, timestamp, attachments, embeds, etc.) to an Apify dataset. - Configurable batch delay to manage Discord API rate limits. - Robust error handling with clear logs for troubleshooting. ## Performance - Execution Time: Fetching 1000 messages takes ~10 seconds (with default 1.0-second batch delay). Fetching the maximum 10,000 messages takes ~100 seconds (~1.7 minutes). - Resource Usage: Optimized for low compute unit consumption, with a maximum limit of 10,000 messages to prevent excessive resource use. ## Input The Actor accepts the following input parameters via the Apify platform: | Key | Type | Description | Default | Required | |-------------------------|--------|------------------------------------------------------------------------------------------------------------------------------|-----------------------------|----------| | guild_id | String | The ID of the Discord guild (server) containing the channel. | - | Yes | | channel_id | String | The ID of the Discord channel to fetch messages from. | - | Yes | | token | String | Your Discord user account token for authentication. Ensure compliance with Discord's Terms of Service. | - | Yes | | max_messages | Integer| Limit the number of messages to fetch (1 to 10,000). Use 100 for quick tests or up to 10,000 for large channels. | 1000 | No | | start_message_id | String | Message ID to start scraping from. For "newer", scrapes messages before this ID; for "older", scrapes messages after this ID.| - | No | | scrape_direction | String | Direction to scrape: "newer" (towards past) or "older" (towards future). | Newer | No | | batch_delay_seconds | String | Delay between batches of 100 messages (non-negative number, in seconds). Increase to 2.0 if rate limit errors occur. | 1.0 | No | ## Output The Actor outputs a dataset in JSON format with the following fields for each product: | Field | Description | |--------------------|----------------------------------------------------| | guild_id | Unique ID of the message. | | channel_id | ID of the channel. | | author | Object with author details (ID, username, discriminator, avatar, public flags). | | content | Message text content. | | timestamp | Creation timestamp (YYYY-MM-DD HH:MM:SS). | | edited_timestamp | Edit timestamp, if edited (YYYY-MM-DD HH:MM:SS). | | type | Message type (e.g., default, reply). | | attachments | Array of attachment objects (ID, filename, size, URL, etc.). | | embeds | Array of embed objects (type, URL, title, description, etc.). | | pinned | Boolean indicating if the message is pinned. | | mention_everyone | Boolean indicating if the message mentions everyone. | | tts | Boolean indicating if the message is text-to-speech. | | flags | Message flags (integer). | | components | Array of message components (e.g., buttons). | | reactions | Array of reaction objects (emoji, count, etc.). | | mentions | Array of mentioned user objects (ID, username, discriminator). | | message_reference| Object with reply details (message ID, channel ID, guild ID), if a reply. | Example Output: json { "message_id": "123456789012345678", "channel_id": "987654321098765432", "author": { "id": "111222333444555666", "username": "user", "discriminator": "1234", "avatar": "https://cdn.discordapp.com/avatars/...", "public_flags": 0 }, "content": "Hello, world!", "timestamp": "2025-10-08 14:30:00", "edited_timestamp": null, "type": 0, "attachments": [], "embeds": [], "pinned": false, "mention_everyone": false, "tts": false, "flags": 0, "components": [], "reactions": [], "mentions": [], "message_reference": null } ## Troubleshooting - Invalid Token: Ensure the token is correct and not expired. Obtain a new token if needed. - Message ID Not Found: Verify the start_message_id exists in the specified channel and is accessible. - Rate Limit Errors: Increase batch_delay_seconds to 2.0 or higher if you see HTTPException errors. - Timeout Errors: If the Actor times out after 60 seconds while fetching the starting message, check the start_message_id and network connection, or contact support for assistance. - No Messages Fetched: Ensure the channel has messages and the token has permission to read the channel’s history.