Google AI Scraper This actor is designed to scrape structured answers from Google's AI-powered search results (AI Overview). Simply provide your questions, and get back comprehensive, well-organized answers with source citations, highlighted key terms, and all answer components—all in a clean, structured JSON format. --- ## 🚀 Key Features - ✅ Extracts AI-generated answers with paragraphs, lists, and headings - ✅ Retrieves source citations with URLs, titles, snippets, and source names - ✅ Captures highlighted key terms from the AI response - ✅ Supports multiple questions in a single run - ✅ Full proxy support for stable and geo-aware scraping - ✅ Handles Google's dynamic content loading seamlessly --- ## 👤 Who Is This Actor For? This actor is perfect for: - 📊 Data analysts gathering AI-curated information at scale - 🧠 Content creators researching topics with authoritative sources - 🔍 SEO professionals analyzing Google's AI-generated content - 📚 Knowledge base builders aggregating information from verified sources - 🤖 AI/ML practitioners collecting training data from Google's AI responses - 🎓 Researchers gathering comprehensive answers with source attribution --- ## ❓ Why Is This Actor Important? Google's AI Overview is a powerful feature that synthesizes information from multiple authoritative sources to answer user questions. The content is dynamically loaded and rendered, making it challenging to access programmatically. This actor: - Waits for the AI response to be fully loaded before extraction - Extracts clean and structured data including paragraphs, lists, headings, and sources - Saves you hours of manual research and data collection - Provides access to Google's AI-curated knowledge at scale --- ## Tech Notes - It is recommended to set the actor memory to at least 4GB to avoid performance issues. - It is recommended to set the actor's proxy type to residential. --- ## Input Schema Here's an example input you can pass to the actor: json { "questions": [ "how many planets are there in the solar system?", "what is photosynthesis?", "who invented the internet?" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ### Input Parameters - questions (required): An array of questions you want answers for. Each question will be submitted to Google's AI search. - proxyConfiguration (required): Proxy settings for the scraper. Essential for avoiding rate limits and bypassing geo-restrictions. --- ## Output Schema json { "question": "how many planets are there in the solar system?", "url": "https://www.google.com/search?udm=50&aep=46&source=25q2-US-SearchSites-Site-CTA&sei=agX6aMCZHYm15NoPl-7zwAQ&q=how%20many%20planets%20are%20there%20in%20the%20solar%20system%3F", "sections": [ { "type": "paragraph", "content": "There are eight planets in our solar system: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. The International Astronomical Union (IAU) officially redefined what constitutes a planet in 2006, leading to Pluto's reclassification as a \"dwarf planet\"." }, { "type": "paragraph", "content": "The eight planets are categorized into two main groups based on their composition:" }, { "type": "list", "listType": "unordered", "items": [ "Inner, terrestrial planets: These four planets are smaller and have solid, rocky surfaces...", "Outer, giant planets: These four are much larger than the terrestrial planets..." ] }, { "type": "heading", "content": "Why Pluto is no longer considered a planet" }, { "type": "list", "listType": "ordered", "items": [ "It must orbit the sun.", "It must be massive enough that its own gravity pulls it into a nearly round shape.", "It must have \"cleared the neighborhood\" around its orbit..." ] } ], "highlightedTerms": ["eight"], "sources": [ { "title": "About the Planets - NASA Science", "snippet": "About the Planets. Our solar system has eight planets: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune...", "sourceName": "NASA Science (.gov)", "url": "https://science.nasa.gov/solar-system/planets/" }, { "title": "Why is Pluto no longer a planet? - The Library of Congress", "snippet": "Answer. The International Astronomical Union (IAU) downgraded the status of Pluto to that of a dwarf planet because it...", "sourceName": "The Library of Congress (.gov)", "url": "https://www.loc.gov/everyday-mysteries/astronomy/item/why-is-pluto-no-longer-a-planet/" } ], "fullText": "There are eight planets in our solar system: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune. The International Astronomical Union (IAU) officially redefined what constitutes a planet in 2006, leading to Pluto's reclassification as a \"dwarf planet\".\n\nThe eight planets are categorized into two main groups based on their composition:\n\n..." } ### Output Fields - question: The question that was asked - sections: Array of answer components (paragraphs, lists, headings) in order - type: Component type (paragraph, list, or heading) - content: Text content for paragraphs and headings - listType: Type of list (ordered or unordered) - items: Array of list items - highlightedTerms: Key terms emphasized in the AI answer - sources: Array of source citations with complete metadata - title: Article or page title - snippet: Preview text from the source - sourceName: Publisher name (e.g., "NASA Science (.gov)") - url: Full URL to the source - fullText: Complete answer text with all sections combined --- ## 💡 Use Cases ### 1. Research & Fact-Checking Get AI-curated answers from authoritative sources for research projects, fact-checking, and knowledge gathering. ### 2. Content Creation Discover comprehensive answers to popular questions with verified sources to inform your content creation strategy. ### 3. SEO & Content Strategy Understand how Google's AI answers questions to optimize your content for AI-powered search results and featured snippets. ### 4. Competitive Intelligence Track how Google's AI presents information about your industry, competitors, or products from various sources. ### 5. Training Data Collection Collect high-quality Q&A pairs with source attribution and highlighted key terms for training language models or building knowledge bases. ### 6. Academic Research Gather structured information with proper source citations for academic papers, literature reviews, or educational content. --- ## 📊 Example Questions Try these question types: - Factual: "How many planets are there in the solar system?" - Scientific: "What is photosynthesis?" - Historical: "Who invented the internet?" - Explanatory: "Why is the sky blue?" - Biographical: "Who was Albert Einstein?" --- 👀 p.s. Got feedback or need an extension? Lexis Solutions is a certified Apify Partner. We can help you with custom solutions or data extraction projects. Contact us over Email or LinkedIn ## Support Our Work 💝 If you're happy with our work and scrapers, you're welcome to leave us a company review here and leave a review for the scrapers you're subscribed to. It will take you less than a minute but it will mean a lot to us! Image Credit: https://www.google.com