Indie Game Market Radar A powerful, multi-platform web scraper that extracts comprehensive metadata from indie game marketplaces including itch.io, Steam, GOG, GameJolt, and Epic Games Store. Perfect for market research, trend analysis, price tracking, and building indie game discovery tools. ## 🎮 Features - Multi-Platform Support: Automatically detects and scrapes from multiple indie game platforms - itch.io - Steam (indie games) - GOG (indie section) - GameJolt - Epic Games Store (indie games) - Rich Metadata Extraction: - Game title and URL - Developer/Publisher information - Price (including currency detection) - Ratings and reviews count - Tags and genres - Release date - Game description - Platform detection - Screenshot URLs - Smart Filtering: Filter games by price range, tags, platform, and more - Intelligent Crawling: Automatically discovers game pages and listing pages - Data Normalization: Consistent data structure across all platforms - Proxy Support: Built-in proxy rotation to prevent blocking ## 🚀 Quick Start ### Input The Actor accepts the following input parameters: json { "startUrls": [ { "url": "https://itch.io/games/tag-indie" } ], "maxRequestsPerCrawl": 100, "extractOnlyGamePages": false, "minPrice": null, "maxPrice": null, "platforms": ["all"], "tags": [] } #### Input Parameters - startUrls (array, required): List of URLs to start crawling from. Can include: - Game listing pages (e.g., https://itch.io/games/tag-indie) - Individual game pages (e.g., https://store.steampowered.com/app/123456) - Search result pages - maxRequestsPerCrawl (integer, default: 100): Maximum number of pages to crawl - extractOnlyGamePages (boolean, default: false): If true, only extracts data from game detail pages, skipping listing pages - minPrice (number, optional): Minimum price filter (in USD) - maxPrice (number, optional): Maximum price filter (in USD) - platforms (array, default: ["all"]): Filter by platform. Options: ["itch.io", "steam", "gog", "gamejolt", "epic", "all"] - tags (array, optional): Filter games by tags (platform-specific) ### Output The Actor outputs structured data to the Apify dataset. Each game entry includes: json { "title": "Hollow Knight", "url": "https://store.steampowered.com/app/367520", "platform": "steam", "developer": "Team Cherry", "publisher": "Team Cherry", "price": 14.99, "currency": "USD", "originalPrice": "$14.99", "discount": null, "rating": 4.8, "reviewsCount": 125000, "tags": ["Metroidvania", "Indie", "Action", "Platformer"], "genres": ["Action", "Adventure"], "releaseDate": "2017-02-24", "description": "Forge your own path in Hollow Knight! An epic action adventure...", "screenshots": [ "https://cdn.akamai.steamstatic.com/steam/apps/367520/ss_123.jpg" ], "thumbnail": "https://cdn.akamai.steamstatic.com/steam/apps/367520/header.jpg" } ## 📊 Use Cases 1. Market Research: Track indie game trends, pricing strategies, and market saturation 2. Competitive Analysis: Monitor competitor games and their metadata 3. Price Tracking: Build price history databases for indie games 4. Discovery Tools: Create indie game recommendation engines 5. Data Aggregation: Build comprehensive indie game databases 6. Trend Analysis: Identify emerging genres, tags, and pricing patterns ## 🛠️ Installation & Usage ### Local Development bash # Install dependencies npm install # Run the Actor npm start # Format code npm run format # Lint code npm run lint ### Apify Platform 1. Push your code to Apify platform 2. Configure input in the Apify Console 3. Run the Actor 4. Download results from the Dataset ## 📋 Supported Platforms ### itch.io - Extracts: title, developer, price, tags, description, ratings - Supports: game pages, tag pages, search results ### Steam - Extracts: title, developer, publisher, price, tags, genres, release date, ratings, reviews - Supports: app pages, indie game listings, search results ### GOG - Extracts: title, developer, price, tags, description, release date - Supports: game pages, indie game listings ### GameJolt - Extracts: title, developer, tags, description, ratings - Supports: game pages, tag pages ### Epic Games Store - Extracts: title, developer, price, tags, description - Supports: game pages, indie game listings ## 🔧 Configuration ### Proxy Configuration The Actor automatically uses Apify's proxy configuration. No additional setup required. ### Rate Limiting The Actor respects robots.txt and includes built-in rate limiting to avoid overwhelming target sites. ## 📈 Output Schema The dataset includes the following fields: | Field | Type | Description | |-------|------|-------------| | title | string | Game title | | url | string | Game page URL | | platform | string | Detected platform (itch.io, steam, gog, etc.) | | developer | string | Developer name | | publisher | string | Publisher name (if different from developer) | | price | number | Price in USD | | currency | string | Currency code (USD, EUR, etc.) | | originalPrice | string | Original price string from page | | discount | number | Discount percentage (if on sale) | | rating | number | Average rating (0-5 scale) | | reviewsCount | number | Number of reviews | | tags | array | Game tags | | genres | array | Game genres | | releaseDate | string | Release date (ISO format) | | description | string | Game description | | screenshots | array | Array of screenshot URLs | | thumbnail | string | Thumbnail image URL | ## 🤝 Contributing This Actor is part of the Apify $1M Challenge. Contributions and improvements are welcome! ## 📝 License ISC ## 🔗 Resources - Apify Documentation - Crawlee Documentation - Apify Store - Apify Challenge ## 💡 Tips - Start with listing pages to discover many games quickly - Use extractOnlyGamePages: true if you only want detailed game data - Filter by platform if you're only interested in specific marketplaces - Adjust maxRequestsPerCrawl based on your needs and rate limits ## 🐛 Troubleshooting Issue: No games extracted - Solution: Check that your start URLs are valid game listing or game detail pages Issue: Missing metadata fields - Solution: Some platforms may not have all fields available. The Actor extracts what's available. Issue: Rate limiting - Solution: Reduce maxRequestsPerCrawl or use Apify's proxy rotation ## 📊 Example Output View the extracted data in the Apify Console's Dataset tab. The data is also available via the Apify API for integration into your applications. --- Built with ❤️ for the indie game community