AI Agent Web Fetcher An advanced web fetcher that can fetch almost all websites and convert them to LLM-friendly Markdown format. Perfect for AI agents, RAG systems, and integration with search actors. ## Features - 🌍 Universal Fetching: Capable of fetching content from almost any website - 🛡️ Advanced Anti-Blocking: Automatically handles complex challenges and protections - 📝 Markdown Output: Clean, LLM-friendly format - 🔗 Integration Ready: Seamlessly integrates with AI Web Search Tool and other search actors - 🎯 Dual Mode: Single URL or batch processing from datasets ## Usage ### Single URL Mode json { "url": "https://example.com" } ### Integration Mode (with AI Web Search Tool) When triggered by the AI Web Search Tool, the actor automatically: 1. Receives the search results dataset 2. Extracts URLs from the link field 3. Fetches and converts each page to Markdown No configuration needed - the integration is automatic! You can also manually specify a dataset: json { "datasetId": "abc123", "urlField": "link", "maxUrls": 10 } ## Input Parameters | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | url | string | Single URL to fetch | - | | datasetId | string | Dataset ID containing URLs (integration mode) | - | | urlField | string | Field name containing URLs in dataset | link | | maxUrls | integer | Max URLs to fetch from dataset (1-100) | 10 | | proxy | object | Proxy configuration | - | | waitUntil | string | Navigation wait condition | domcontentloaded | | debug | boolean | Save HTML content in output | false | ## Output Format Each fetched URL produces: json { "url": "https://example.com", "markdown": "# Page Title\n\nContent...", "metadata": { "title": "Page Title", "description": "Page description", "author": "Author name" } } ## Integration with AI Web Search Tool 1. Run the AI Web Search Tool 2. Add this actor as an integration 3. The actor automatically fetches and converts all search results to Markdown Perfect for building AI agents that need to: - Research topics with real-time web data - Extract content from search results - Build RAG systems with fresh web content ## Standby Mode (HTTP API) This Actor supports Standby mode for real-time HTTP API access with low latency. When enabled, the Actor runs as a persistent HTTP server. ### Base URL https://<your-username>--ai-fetch-python.apify.actor ### Endpoints | Endpoint | Method | Description | |----------|--------|-------------| | / | GET | API information and available endpoints | | /health | GET | Health check endpoint | | /fetch | GET/POST | Fetch a URL and return Markdown | ### Fetch Parameters | Parameter | Type | Description | |-----------|------|-------------| | url | string | URL to fetch (required) | | waitUntil | string | Wait condition: networkidle, load, domcontentloaded (default: networkidle) | ### Example Requests Fetch a URL (GET): bash curl "https://<username>--ai-fetch-python.apify.actor/fetch?url=https://example.com&token=YOUR_TOKEN" Fetch with POST: bash curl -X POST "https://<username>--ai-fetch-python.apify.actor/fetch?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "waitUntil": "networkidle"}' ### Response Format json { "url": "https://example.com", "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...", "metadata": { "title": "Example Domain", "description": "", "author": "" } }