arXiv API & Full-Text Scraper (AI-Ready) This Apify Actor provides a complete, AI-ready dataset from arXiv.org. It uses the official arXiv API for fast and reliable metadata scraping, then downloads, cleans, and chunks the full text from each paper's PDF. This tool is designed for AI/LLM developers, researchers, and data scientists who need high-quality text corpora for model training and Retrieval-Augmented Generation (RAG) pipelines. ## Key Competitive Features * ⚡️ Blazing Fast & Low Cost: Uses the official arXiv API, not a slow browser, to find papers. This is thousands of times faster and cheaper than other scrapers. * 🤖 AI-Ready Chunking: Automatically splits the clean text into overlapping chunks, perfect for ingestion into vector databases (RAG). * 🧹 Automatic Text Cleaning: Cleans the raw PDF text to remove headers, footers, page numbers, and bibliographies. * 🎯 Powerful Search: Scrape by keyword, category code (e.g., cs.AI), or both. --- ## Input The actor accepts the following JSON input. searchQuery or category is required. | Field | Type | Description | Default | | :--- | :--- | :--- | :--- | | searchQuery | String | (Optional) The keyword search query (e.g., "black hole physics"). | | | category | String | (Optional) The arXiv category code (e.g., cs.AI for AI, gr-qc for General Relativity). | | | maxPages | Number | The maximum number of result pages to scrape (50 results per page). | 1 | | chunkSize | Number | (Optional) The target size (in characters) for each text chunk. | 1000 | | chunkOverlap| Number | (Optional) The number of characters to overlap between chunks. | 200 | ### Example Input (Simple Keyword Search) json { "searchQuery": "quantum computing", "maxPages": 3 } ### Example Input (AI-Ready Category Search)json { "category": "cs.CL", "maxPages": 10, "chunkSize": 1500, "chunkOverlap": 300 } ### Example Output { "title": "Quantum black holes: inside and outside", "authors": "Bernard S. Kay", "abstract": "We review and add to a conjectured 'principle' according to which...", "url": "http://arxiv.org/abs/2510.20799v1", "cleanedFullText": "Quantum black holes: inside and outside\n\nBernard S. Kay\n\n(Dated: October 23, 2025)\n\nAbstract\nWe review and add to a conjectured 'principle' according to which...\n\n[... rest of the cleaned paper text ...]\n\n", "textChunks": [ "Quantum black holes: inside and outside\n\nBernard S. Kay\n\n(Dated: October 23, 2025)\n\nAbstract\nWe review and add to a conjectured 'principle' according to which a necessary condition for a ... [900 more characters]", "necessary condition for a ... [800 more characters] ... The principle is intended to apply only to 'civilized' spacetimes. We shall recall a ... [and 200 more characters]", "[... etc ...]" ] }