Twitter Get UserByScreenName V2

Overview

This Apify Actor scrapes a single Twitter/X user profile page to extract data. It takes a target profile URL as input, fetches the HTML using Axios, parses it with Cheerio, and outputs structured user data to a dataset. It's built on Apify's single-page JavaScript scraping template.

Key Features

• Apify SDK: Provides the foundation for building and running the Actor.
• Input Schema: Validates the required input (the target Twitter profile URL).
• Dataset Storage: Outputs structured, consistent JSON data for easy retrieval.
• Axios & Cheerio: Uses Axios for reliable HTTP requests and Cheerio for fast HTML parsing and data extraction.
• Customizable: The parsing logic can be edited to extract different data points from the profile page.

Input/Output

Input
The Actor requires a JSON input containing the URL of the Twitter/X user profile to scrape.

{
  "url": "https://twitter.com/username"
}

Output
The scraped data is saved to an Apify dataset. By default, the template extracts page headings (h1-h6), but this Actor is configured to parse specific user profile information (like display name, handle, bio, follower count, etc.). Each item in the dataset is a JSON object containing the extracted fields.

How to Use

1. Configure Input: In the Actor's console, provide the target Twitter profile URL in the input field.
2. Run the Actor: Start the run. The Actor will fetch the page, parse it, and store the results.
3. Get Results: After the run finishes, access the scraped user data from the Dataset tab in the console. You can view it as JSON, CSV, or via API.

Local Development

To modify or run the Actor locally:

1. Install the Apify CLI:
   bash npm -g install apify-cli
2. Pull the Actor to your machine using its unique name or ID:
   bash apify pull <ActorId>
3. Edit the parsing logic in main.js (particularly the Cheerio selectors) to tailor the data extraction.

How It Works (Code Flow)

1. Actor.getInput() retrieves the input configuration, including the target URL.
2. axios.get(url) fetches the HTML of the Twitter profile page.
3. cheerio.load(response.data) loads the HTML into a Cheerio instance for parsing.
4. The script uses Cheerio selectors (e.g., $('div[data-testid="UserName"]')) to find and extract specific user profile elements.
5. Actor.pushData(userData) saves the structured extracted data to the dataset.

Resources

• Apify SDK for JavaScript documentation
• Apify Platform documentation
• Web scraping in Node.js with Axios and Cheerio
• Join our developer community on Discord