AI Agent Search

A browser-based content extraction tool built on Apify's Puppeteer Crawler. It loads web pages with full JavaScript rendering and extracts both visible text content and structured data, designed primarily for feeding real-time website information to AI agents.

Key Features

• Full JavaScript Rendering: Uses Puppeteer and Chrome to load pages exactly like a real browser, ensuring dynamic content is captured.
• Dual Content Extraction: Retrieves both the main visible text content and structured data from each page.
• Optional Screenshots: Can capture screenshots of the loaded pages.
• Configurable Crawling: Allows you to set crawl depth and a maximum number of pages to process.
• Apify Platform Integration: Runs seamlessly as an Apify actor.

How to Use

The actor is configured via a JSON input. The primary requirement is to provide one or more startUrls. Each URL object should specify the url and the HTTP method (typically "GET").

Input/Output

Input Example:

{
  "startUrls": [
    { "url": "https://openrouter.ai/", "method": "GET" },
    { "url": "https://discord.com", "method": "GET" }
  ]
}

Output Example:
The actor returns an array of objects, one for each processed page. Each object contains the page's URL, title, and the extracted text content.

[
  {
    "url": "https://openrouter.ai/",
    "title": "OpenRouter",
    "content": "Skip to content OpenRouter / Models Chat Rankings Docs Sign in The Unified Interface For LLMs Better prices , better uptime , no subscription. Featured Models View Trending Qwen3 235B A22B by qwen 13.7B Tokens/wk 983ms Latency +29.32% Weekly growth GPT-4.1 by openai 33.1B Tokens/wk 628ms Latency -4.13% Weekly growth Claude 3.7 Sonnet by anthropic 273.9B Tokens/wk 1.8s Latency -4.51% Weekly growth 7.9T Monthly Tokens 2M Global Users 50+ Active Providers 300+ Models..."
  }
]

This provides AI agents with the publicly visible data from a website, giving them context to work with. It's a straightforward tool for fetching live web content for automated processing.