OSINT Scraper

Overview

An Apify actor that scrapes multiple public paste and code-sharing websites for potentially sensitive data based on your search terms. It's designed for Open Source Intelligence (OSINT) gathering.

Key Features

• Keyword Search: Scrape content using any custom keywords.
• Multi-Site Support: Targets sites like Pastebin, GitHub Gist, Codepad, Ideone, Paste.org, and Textbin.
• Modular: Enable or disable scraping for specific websites via input flags.
• Extensible: Allows custom data extraction via a JavaScript function.
• Efficient: Optimized for speed with low compute unit consumption (~0.01-0.03 units per 100 pages).

How to Use

The actor requires a JSON input configuration. You must use a proxy; you can use your own or Apify Proxy.

Tip: To scrape only specific sites, set their corresponding flags to true in the input. For Pastebin, US-based proxies are recommended due to regional restrictions.

Input

Required input is a JSON object. The searchKeywords array and proxy configuration are mandatory.

{
  "searchKeywords": ["@gmail", "db_pass"],
  "codepad": true,
  "githubgist": true,
  "ideone": true,
  "pastebin": true,
  "pasteorg": true,
  "textbin": true,
  "proxy": {
    "useApifyProxy": true
  },
  "extendOutputFunction": "($) => { return {'customField': $('title').text()} }"
}

Input Fields

• searchKeywords: (Required) Array of strings containing keywords to search for.
• proxy: (Required) Proxy configuration object.
• codepad, githubgist, ideone, pastebin, pasteorg, textbin: (Optional) Boolean flags to enable/disable scraping for each specific website.
• extendOutputFunction: (Optional) A string containing a JavaScript function for custom data extraction. The function receives a JQuery handle ($) as an argument.

Output

Results are stored in the Apify dataset. Each item represents a found match and has the following structure:

{
  "keyword": "a",
  "url": "https://gist.github.com/trin94/3381395adc8b2c3fea81a38b9a385369"
}

You can manage results using the Apify API in Python, PHP, Node.js, or other languages. See the Apify API reference for details.

Development & Support

This actor is under active development. For bug reports or feature requests, create an issue on the GitHub repository. For more information, visit epctex.com.