Dynamic Markdown Scraper

A powerful web scraper that converts difficult to scrape web pages into clean, well-formatted Markdown content. This scraper crawls websites and automatically transforms their HTML content into Markdown format while maintaining the original structure and formatting. It handles dynamic content and JavaScript-rendered pages with ease. ## Features - Crawls websites and converts content to Markdown format - Maintains proper heading structure, lists, and code blocks - Handles dynamic content and JavaScript-rendered pages - Handles images and links correctly - Respects same-domain crawling - Filters out unwanted content (navigation, footers, etc.) - Configurable maximum crawl limits - Smart content extraction focusing on main article content - Built with TypeScript for better maintainability ## Use Cases - Feed website content to LLM AI for further processing - Extract content from websites for documentation, blog posts, or technical writing - Scrape and convert web pages for use in static sites, blogs, or other projects - Automate content migration from legacy systems to modern platforms ## Input Configuration The scraper accepts the following input parameters: - startUrls: Array of URLs where the crawler should begin (required) - maxRequestsPerCrawl: Maximum number of pages to crawl (optional, defaults to unlimited) Example input: json { "startUrls": [ { "url": "https://apify.com" } ], "maxRequestsPerCrawl": 100 } ## Output Format The scraper saves the following data for each processed page: - url: The URL of the scraped page - title: Page title - markdown: Converted Markdown content - capturedAt: Timestamp of when the page was scraped Example output: json { "url": "https://apify.com/storage", "title": "Storage optimized for scraping · Apify", "markdown": "# Apify Storage\n\nScalable and reliable cloud data storage designed for web scraping and automation workloads.\n\n[View documentation](https://docs.apify.com/platform/storage)\n\nBenefits\n\n## Specialized storage from Apify[](https://apify.com/storage#specialized-storage-from-apify)\n\n![Enterprise_grade_reliability_performance_and_scalability_9890860f85.svg](https://cdn-cms.apify.com/Enterprise_grade_reliability_performance_and_scalability_9890860f85.svg)\n\n### Enterprise-grade reliability, performance, and scalability[](https://apify.com/storage#enterprise-grade-reliability-performance-and-scalability)\n\nStore a few records or a few hundred million, with the same low latency and high reliability. We use Amazon Web Services for the underlying data storage, giving you high availability and peace of mind.\n\n### Low-cost storage for web scraping and crawling[](https://apify.com/storage#low-cost-storage-for-web-scraping-and-crawling)\n\nApify provides low-cost storage carefully designed for the large workloads typical of web scraping and crawling operations.\n\n![Low_cost_storage_for_web_scraping_and_crawling_b313f7d95e.svg](https://cdn-cms.apify.com/Low_cost_storage_for_web_scraping_and_crawling_b313f7d95e.svg)\n\n![Easy_to_use_634e40ae76.svg](https://cdn-cms.apify.com/Easy_to_use_634e40ae76.svg)\n\n### Easy to use[](https://apify.com/storage#easy-to-use)\n\nData can be viewed on the web, giving you a quick way to review and share it with other people. The Apify [API](https://docs.apify.com/api/v2) and [SDK](https://docs.apify.com/sdk/js/) makes it easy to integrate our storage into your apps.\n\nFeatures\n\n## We’ve got you covered[](https://apify.com/storage#weve-got-you-covered)\n\n[![Dataset_78dfe4e3a4.svg](https://cdn-cms.apify.com/Dataset_78dfe4e3a4.svg)\n\n**Dataset** \nStore results from your web scraping, crawling or data processing jobs into Apify datasets and export them to various formats like JSON, CSV, XML, RSS, Excel or HTML.\n\n\n\n\n\n](https://docs.apify.com/platform/storage/dataset)[![Request_queue_9e9602319e.svg](https://cdn-cms.apify.com/Request_queue_9e9602319e.svg)\n\n**Request queue** \nMaintain a queue of URLs of web pages in order to recursively crawl websites, starting from initial URLs and adding new links as they are found while skipping duplicates.\n\n\n\n\n\n](https://docs.apify.com/platform/storage/request-queue)[![Key_value_store_bc65220b7d.svg](https://cdn-cms.apify.com/Key_value_store_bc65220b7d.svg)\n\n**Key-value store** \nStore arbitrary data records along with their MIME content type. The records are accessible under a unique name and can be written and read at a rapid rate.\n\n\n\n\n\n](https://docs.apify.com/platform/storage/key-value-store)\n\n## Ready to build your first Actor?[](https://apify.com/storage#ready-to-build-your-first-actor)\n\n[Start developing](https://apify.com/templates)", "capturedAt": "2025-01-23T14:01:21.956Z" }