🗺️ Sitemap Generator (Apify Actor) Generate a clean, standards-compliant sitemap.xml for a website — automatically, reliably, and without manual cleanup. This actor crawls a single website, discovers all indexable pages, and produces: - ✅ A ready-to-submit sitemap.xml (Google-compliant) - ✅ A structured JSON dataset of discovered URLs (for auditing, reporting, and billing) Built for SEO professionals, agencies, and site owners who want accuracy, transparency, and results they can trust. ## ✅ What This Actor Does - Crawls one website per run (no mixed domains, no confusion) - Discovers internal pages by following links - Excludes junk/system URLs automatically (e.g. Cloudflare, admin endpoints) - Respects robots.txt (optional) - Removes duplicate URLs and URL fragments - Optionally strips query strings to prevent sitemap bloat - Extracts real <lastmod> dates when available: - From HTTP Last-Modified headers - From blog/article meta tags when headers are missing - Outputs a fully valid sitemap.xml ## 📦 Outputs (Where to Find Your Files) Run → Storage → Key-value store → sitemap.xml This file is: - Ready to upload to Google Search Console - Ready to host at /sitemap.xml - Standards-compliant (no reconstruction required) ### 🟢 JSON Results (Dataset) Every discovered page is also saved to the Dataset. Each row includes: - url – discovered page URL - depth – crawl depth from the homepage - lastmod – modification date (when available) - lastmodSource – "header", "meta", or null This dataset is useful for: - Auditing and QA - URL counts and reporting - Monetization and billing logic - Previewing results before download ## 🔒 Important Design Decisions (On Purpose) ### One Website per Run This actor enforces a single start URL. Why? - A sitemap must not mix domains - One site = one sitemap = one clean result - Prevents invalid or rejected sitemaps - Enables clear pricing per site ### Honest <lastmod> Values The actor does not fake modification dates. - Uses real server headers when available - Falls back to article metadata for blog posts - Omits <lastmod> when no trustworthy source exists This avoids misleading search engines and protects SEO integrity. ## ⚙️ Inputs ### Required - Start URL The root URL of the website (example: https://example.com) ### Optional - Max crawl depth - Max number of pages - Concurrency - Headless browser (for JavaScript-heavy sites) - Strip query strings - Respect robots.txt - Advanced include/exclude URL patterns (regex) Most users can run the actor with just a Start URL. ## 🧠 Who This Is For - SEO professionals - Agencies managing multiple client sites - Developers who need clean sitemaps programmatically - Site owners preparing for Google Search Console - AI-first websites optimizing crawlability ## 💡 Why Use This Actor Instead of Online Sitemap Tools? - No URL limits - No fake results - No mixed domains - No guessing which pages were included - Full transparency (XML + JSON) - Automation-ready and API-friendly ## 🔐 PPE (Paid / Private / Enterprise) This actor is designed for PPE use: - Consistent, auditable outputs - Dataset always populated (even if XML is downloaded) - Clear value per run - Suitable for client-facing and internal workflows Run it. Download sitemap.xml. Submit. Done. ### 🟢 sitemap.xml (Primary Output) Your sitemap is written as a real XML file. Location in Apify UI: