Sitemap Finder A powerful Apify Actor that finds sitemap URLs for any website. This Actor helps you discover XML sitemaps by checking common locations, robots.txt files, and analyzing HTML content for sitemap links. ## Features - Checks common sitemap locations (e.g., /sitemap.xml, /sitemap-index.xml, etc.) - Parses robots.txt for Sitemap directives - Analyzes website HTML for sitemap links - Verifies discovered URLs are valid XML sitemaps - Option to find either the primary sitemap or all available sitemaps - Configurable request timeout and verbose logging ## Input Parameters | Parameter | Type | Description | |-----------|------|-------------| | url | String | The URL of the website to search for sitemaps (required) | | findAll | Boolean | If true, find all available sitemaps. If false, find only the primary sitemap (default: true) | | noVerify | Boolean | If true, skip verification of sitemap XML format (default: false) | | timeout | Integer | Timeout in seconds for HTTP requests (default: 5) | | verbose | Boolean | Enable detailed logging of the sitemap discovery process (default: false) | ## Output The Actor saves results to the default dataset with the following structure: ### When findAll = true json { "url": "example.com", "sitemaps": [ "https://example.com/sitemap.xml", "https://example.com/post-sitemap.xml" ], "count": 2 } ### When findAll = false json { "url": "example.com", "sitemap": "https://example.com/sitemap.xml" } ## Example Usage You can run the Actor on the Apify Platform or integrate it into your own applications using the Apify API. ### Running on Apify Platform 1. Navigate to the Actor page 2. Enter the website URL you want to search for sitemaps 3. Configure optional parameters as needed 4. Run the Actor and wait for results ### API Example python import apify_client # Initialize the ApifyClient with your API token client = apify_client.ApifyClient(token="YOUR_API_TOKEN") # Prepare the Actor input run_input = { "url": "example.com", "findAll": True, "verbose": True } # Run the Actor and wait for it to finish run = client.actor("YOUR_ACTOR_ID").call(run_input=run_input) # Fetch and print Actor results from the default dataset results = client.dataset(run["defaultDatasetId"]).list_items().items print(results) ## Use Cases - SEO Analysis: Quickly locate all sitemaps to analyze website structure and content organization - Web Scraping: Find sitemaps to extract URLs for crawling - Website Auditing: Check if sitemaps are properly configured and accessible - Content Discovery: Identify pages and sections included in sitemaps ## Resources - Apify SDK for Python documentation - Apify Platform documentation - Join our developer community on Discord