LinkedIn Profile Posts Scraper Actor 🚀 Apify Actor to scrape ALL posts from a LinkedIn profile via n8n webhook This actor extracts comprehensive data from all posts published by a LinkedIn profile by calling your n8n webhook endpoint. It's designed to work seamlessly with your existing n8n workflow for LinkedIn profile posts extraction. ## 🎯 Features - ✅ Scrapes ALL posts from a LinkedIn profile (not just one) - ✅ Configurable maximum posts limit (default: 50, max: 200) - ✅ Extracts complete post data (title, content, author, engagement) - ✅ Retrieves author information (name, profile URL, followers) - ✅ Captures engagement metrics (reactions, comments) - ✅ Extracts hashtags from post content - ✅ Gets media/images from posts - ✅ Retrieves top comments for each post - ✅ Captures published date - ✅ Multiple retry attempts with different user agents - ✅ Comprehensive error handling and logging - ✅ Dataset views for easy data visualization - ✅ Aggregate statistics (total reactions, comments) ## 📥 Input json { "profileUrl": "https://www.linkedin.com/in/cloudwithraj/", "maxPosts": 50 } ### Input Parameters | Field | Type | Required | Description | |-------|------|----------|-------------| | profileUrl | String | Yes | Full URL of the LinkedIn profile to scrape posts from | | maxPosts | Integer | No | Maximum number of posts to scrape (default: 50, max: 200) | ## 📤 Output The actor returns an array of posts scraped from the profile: json [ { "postIndex": 1, "activityId": "7388952726245629952", "url": "https://www.linkedin.com/posts/cloudwithraj_...", "title": "Post title", "author": { "name": "Author Name", "url": "https://www.linkedin.com/in/authorname", "followers": "10K followers" }, "content": { "text": "Post content text...", "hashtags": ["#amazon", "#tech", "#news"] }, "engagement": { "reactions": 1234, "comments": 56 }, "media": { "images": ["https://media.licdn.com/..."] }, "publishedDate": "2024-12-01T10:30:00.000Z", "topComments": [ { "author": "Commenter Name", "text": "Comment text...", "time": "2h ago" } ], "scrapedAt": "2024-12-01T15:45:00.000Z", "source": "linkedin-profile-posts-scraper", "profileUrl": "https://www.linkedin.com/in/cloudwithraj/", "username": "cloudwithraj" }, { "postIndex": 2, ... } ] ## 🎨 Dataset Views The actor provides 3 pre-configured dataset views: ### 1. Overview Basic information for quick analysis: - Post Index (#) - Activity ID - Title - Author - Post text - Reactions & Comments - Published date - Post URL ### 2. Detailed View Complete post information including: - All overview fields - Author profile URL and followers - Hashtags - Media/images - Top comments - Scraping metadata - Profile URL and username ### 3. Engagement Metrics Focus on engagement data: - Post Index - Activity ID - Author - Reactions & Comments count - Hashtags - Published date ## 🚀 Usage ### On Apify Platform 1. Go to Apify Console 2. Create a new Actor 3. Upload all files from this directory 4. Build and run the actor 5. Provide a LinkedIn post URL in the input ### Local Development bash # Install Apify CLI npm install -g apify-cli # Initialize project (if not already) apify init # Run locally apify run -p # Push to Apify platform apify push ### Via API bash curl -X POST https://api.apify.com/v2/acts/YOUR_ACTOR_ID/runs \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "profileUrl": "https://www.linkedin.com/in/username/", "maxPosts": 50 }' ## 🔧 Technical Details ### Architecture - Webhook Integration: Calls n8n webhook at https://eliasse-n8n.onrender.com/webhook/1305b7f8-0602-48f6-85df-0f7f7497e54b - HTTP Client: Uses axios with retry logic - Error Handling: Multiple retry attempts with different user agents - Timeout: 180 seconds per request (3 minutes - profile scraping takes longer) - Data Validation: Only includes non-empty fields in output - Bulk Processing: Handles multiple posts from a single profile ### Webhook Payload json { "profileUrl": "https://www.linkedin.com/in/username/", "username": "username", "maxPosts": 50, "timestamp": "2024-12-01T15:45:00.000Z", "source": "linkedin-profile-posts-scraper", "version": "1.0.0", "requestType": "profile_posts_scraping" } ### Retry Logic The actor implements smart retry logic: 1. Tests multiple payload formats 2. Tries different user agents 3. Maximum 3 attempts per configuration 4. 2-second delay between retries ## ⚠️ Limitations - Requires valid LinkedIn profile URL (linkedin.com/in/username) - Profile posts must be publicly accessible (or semi-public) - Depends on n8n webhook availability - Maximum 200 posts per profile (configurable) - Some data may not be available for all posts - Rate limiting may apply - Scraping time increases with number of posts ## 📊 Data Quality The actor ensures data quality by: - Validating URL format before processing - Filtering out empty/null/"Not specified" values - Providing structured nested data (author, content, engagement, media) - Including metadata for tracking (scrapedAt, source, postIndex) - Aggregate statistics (total reactions, comments across all posts) - Post indexing for easy reference ## 🔐 Privacy & Ethics - Only scrapes publicly accessible posts - Respects LinkedIn's robot.txt and ToS - No authentication required - Data is stored securely in Apify datasets ## 🐛 Error Handling The actor handles various error scenarios: - Invalid URL format → Returns error with helpful message - Webhook timeout → Retries with exponential backoff - Network errors → Logs detailed error info - Empty response → Returns error item to dataset ## 📝 Logs The actor provides detailed logging: 👤 LinkedIn Profile Posts Scraper Actor Starting... 📝 Input received: {...} 🔍 Scraping posts for profile: cloudwithraj (max: 50 posts) 🔄 Attempt 1: Testing payload format ✅ Success with payload format and user agent: Mozilla/5.0... 📊 Received data for processing: 12456 characters 📰 Processing 25 posts from profile ✅ Processed 25 posts from profile 📋 Sample post: {...} ✅ Saved 25 posts to dataset 📊 Final Summary: { "profileUrl": "https://www.linkedin.com/in/cloudwithraj/", "username": "cloudwithraj", "totalPostsScraped": 25, "postsWithContent": 25, "postsWithEngagement": 23, "totalReactions": 15234, "totalComments": 456, "errors": 0, "maxPosts": 50 } 🏁 LinkedIn Profile Posts Scraper Actor Finished ## 📈 Version History ### v1.0.0 (2024-12-01) - Initial release - Profile posts scraping functionality (all posts from a profile) - Configurable max posts limit (1-200) - Webhook integration with n8n - Multiple dataset views - Comprehensive error handling - Aggregate statistics ## 🤝 Support For issues or questions: 1. Check the actor logs in Apify Console 2. Verify the profile URL is valid and accessible (linkedin.com/in/username) 3. Ensure n8n webhook is online and responding 4. Check if profile has public posts available --- Made with ❤️ for LinkedIn profile data extraction