10times - Top 100 Pages Crawler ## Overview This crawler extracts event data from 10times Top 100 pages across multiple categories. It provides comprehensive information about events including their rank, name, dates, location, category, and rating. The crawler supports all major industry categories: Agriculture & Forestry, Animals & Pets, Apparel & Clothing, Arts & Crafts, Auto & Automotive, Baby, Kids & Maternity, Banking & Finance, Building & Construction, Business Services, Education & Training, Electric & Electronics, Entertainment & Media, Environment & Waste, Fashion & Beauty, Food & Beverages, Home & Office, Hospitality, IT & Technology, Industrial Engineering, Logistics & Transportation, Medical & Pharma, Miscellaneous, Packing & Packaging, Power & Energy, Science & Research, Security & Defense, Telecommunication, Travel & Tourism, Wellness, Health & Fitness. ## Features - Multi-Category Support: Extracts data from 30+ industry categories - Comprehensive Event Details: Collects event names, dates, locations, and ratings - Structured Output: Returns data in an organized, easy-to-use JSON format - Configurable Input: Works with any category from the 10times Top 100 pages - Advanced Anti-Detection: Implements sophisticated stealth techniques to avoid blocking - Robust Error Handling: Comprehensive error handling and retry mechanisms - Session Management: Intelligent session rotation and cookie management ## Input Parameters | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | url | String | The category URL to scrape | https://10times.com/top100/travel-tourism | | proxyConfiguration | Object | Proxy configuration settings | See below | | maxRetries | Integer | Maximum retry attempts (1-10) | 5 | | requestTimeout | Integer | Request timeout in seconds (30-600) | 300 | ## Example Input json { "url": "https://10times.com/top100/travel-tourism", "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "" }, "maxRetries": 5, "requestTimeout": 300 } ## Proxy Configuration IMPORTANT: Proper proxy configuration is crucial for successful scraping. 10times.com has sophisticated anti-bot measures. ### Recommended Settings: json { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "IN" } ### Why These Settings Work Best: 1. RESIDENTIAL Proxy Group: Provides real residential IP addresses that are less likely to be blocked 2. India (IN) Country: 10times.com has better accessibility from Indian IP addresses 3. Session Rotation: The crawler automatically rotates sessions to avoid detection ### Alternative Proxy Options: - DATACENTER: Faster but more likely to be blocked - RESIDENTIAL: Slower but more reliable (recommended) - Other Countries: Try US, UK, or DE if IN doesn't work ## Output Format The crawler produces structured JSON data with the following format: json [ { "rank": "1", "eventLink": "https://10times.com/magic", "eventName": "MAGIC LAS VEGAS", "eventDates": "10 - 12 Feb 2025", "dateStart": "20250210T000000Z", "dateEnd": "20250212T235959Z", "location": "Las Vegas", "category": "Apparel & Clothing", "rating": "4.3", "extractedAt": "2025-01-15T10:30:00.000Z", "sourceUrl": "https://10times.com/top100/travel-tourism" } ] ## Available Categories This crawler supports all categories listed on https://10times.com/top100/, including: - Agriculture & Forestry - Animals & Pets - Apparel & Clothing - Arts & Crafts - Auto & Automotive - Baby, Kids & Maternity - Banking & Finance - Building & Construction - Business Services - Education & Training - Electric & Electronics - Entertainment & Media - Environment & Waste - Fashion & Beauty - Food & Beverages - Home & Office - Hospitality - IT & Technology - Industrial Engineering - Logistics & Transportation - Medical & Pharma - Miscellaneous - Packing & Packaging - Power & Energy - Science & Research - Security & Defense - Telecommunication - Travel & Tourism - Wellness, Health & Fitness ## Troubleshooting ### Common Issues and Solutions #### 1. 403 Forbidden Errors Symptoms: - Request blocked - received 403 status code - Access denied - page blocked Solutions: - Use RESIDENTIAL proxy group instead of DATACENTER - Set proxy country to India (IN) - Increase delays between requests - Check if the URL is valid and accessible #### 2. Execution Context Destroyed Symptoms: - page.evaluate: Execution context was destroyed, most likely because of a navigation Solutions: - This is usually caused by page redirects or anti-bot measures - The crawler automatically handles this with retry logic - Ensure proper proxy configuration #### 3. Tunnel Connection Failed Symptoms: - net::ERR_TUNNEL_CONNECTION_FAILED Solutions: - Check proxy configuration - Try different proxy groups or countries - Ensure sufficient proxy credits #### 4. No Data Extracted Symptoms: - No valid data extracted - Event list table not found Solutions: - Verify the URL is correct and accessible - Check if the page structure has changed - Use different proxy settings - Try a different category URL #### 5. Session Rotation Issues Symptoms: - Multiple failed requests with the same session - Sessions being marked as bad repeatedly Solutions: - The crawler automatically handles session rotation - Check proxy configuration - Ensure sufficient proxy credits ### Debug Information The crawler generates debug files when issues occur: - debug-{timestamp}.png: Screenshots when table not found - error-{timestamp}.png: Screenshots when errors occur These files help identify what the page looks like when issues occur. ### Performance Optimization 1. Use RESIDENTIAL Proxies: More reliable but slower 2. Limit Concurrency: Keep at 1 for best results 3. Proper Delays: Built-in random delays prevent detection 4. Session Management: Automatic session rotation ## Technical Details ### Anti-Detection Features - User Agent Rotation: Random selection from realistic browser user agents - Viewport Randomization: Different screen sizes for each request - Stealth Script Injection: Masks automation indicators - Human-like Behavior: Random mouse movements and scrolling - Cookie Management: Preserves successful session cookies - Header Randomization: Realistic HTTP headers ### Error Handling - Automatic Retries: Configurable retry attempts - Session Rotation: Bad sessions are automatically replaced - Timeout Management: Configurable timeouts for different operations - Graceful Degradation: Continues processing even if some requests fail ## Legal Notice By using this actor, you agree to: 1. Respect Website Terms: Comply with the terms of service of 10times.com 2. Data Ownership: Acknowledge that the data belongs to respective owners and is subject to copyright laws 3. Responsible Use: Use the actor only for lawful purposes without causing harm to the target website 4. Privacy Compliance: Handle any collected data in accordance with applicable privacy regulations 5. Liability: Accept responsibility for how the data is used and any consequences thereof ## Support If you encounter persistent issues: 1. Check the troubleshooting section above 2. Verify your proxy configuration 3. Try different category URLs 4. Ensure you have sufficient proxy credits 5. Check the debug screenshots for visual clues The crawler is designed to be robust and handle most common issues automatically, but proper proxy configuration is essential for success.