SlashDot Technology News Scraper This Apify actor scrapes technology news articles from SlashDot.org, extracting comprehensive information about articles, their content, engagement metrics, and community discussions. ## Features - Comprehensive Article Data: Scrapes detailed information about technology news articles - Content Analysis: Extracts full article content, summaries, and metadata - Engagement Metrics: Collects comment counts, scores, views, and ratings - Community Features: Gathers comments, discussions, and user interactions - Categorization: Extracts sections, tags, and topic classifications - Related Content: Finds related articles and cross-references - Filtering Options: Supports filtering by sections and sorting methods - HTML Debugging: Saves HTML content for selector analysis during development ## Input Parameters | Parameter | Type | Default | Description | | --------------- | ------- | -------- | --------------------------------------------- | | maxArticles | Integer | 100 | Maximum number of articles to scrape | | scrapeDetails | Boolean | true | Whether to scrape detailed article pages | | sections | Array | [] | List of sections to filter by | | sortBy | String | "latest" | Sort method (latest, popular, most_commented) | ## Output Data Each article record includes: ### Basic Information - article_id: Unique article identifier - title: Article title - summary: Article summary/teaser - url: URL to the full article - image_url: Article thumbnail/preview image URL ### Author and Publication - author: Article author name - published_date: When the article was published - section: Article section/category ### Categorization - tags: Array of tags and labels ### Engagement Metrics - comment_count: Number of comments - score: Article score/rating - views: Number of views ### Timestamps - scraped_at: When the data was scraped ### Detailed Information (if scrapeDetails=true) - full_content: Complete article content - paragraphs: Array of article paragraphs - related_articles: Array of related articles with title and URL - comments: Array of comments with text, author, date, and score - media_files: Array of media files with URL, type, and alt text - source_links: Array of external source links - metadata: Article metadata from meta tags ### Metadata - source: Source website (slashdot.org) ## Usage Examples ### Basic Usage json { "maxArticles": 50, "scrapeDetails": true } ### Filtered by Section json { "maxArticles": 200, "scrapeDetails": true, "sections": ["technology", "science"], "sortBy": "popular" } ### Most Commented Articles json { "maxArticles": 100, "scrapeDetails": true, "sortBy": "most_commented" } ### Quick Scraping (No Details) json { "maxArticles": 500, "scrapeDetails": false, "sortBy": "latest" } ## Development Features ### HTML Debugging During development, the scraper saves HTML content to the key-value store for selector analysis: - debug_slashdot_html: Contains the HTML content of the main page ### Error Handling - Comprehensive error handling with detailed logging - Graceful handling of missing elements - Retry logic for failed requests ### Browser Automation - Uses Playwright for reliable browser automation - Handles dynamic content loading - Implements proper delays and waits ## Installation 1. Install dependencies: bash pip install -r requirements.txt 2. Install Playwright browsers: bash playwright install chromium 3. Run the scraper: bash python -m src ## Docker Usage bash docker build -t slashdot-scraper . docker run -e APIFY_TOKEN=your_token slashdot-scraper ## Notes - The scraper respects rate limits and implements delays between requests - HTML content is saved for debugging purposes during development - The scraper handles various article listing layouts and structures - All URLs are properly resolved and normalized - Comment extraction includes author information and engagement metrics - The scraper can handle both article listings and detailed article pages