NPM Scraper

Overview

This actor scrapes data from npmjs.com, providing an alternative to npm's limited official API. It can retrieve package details, search results, and user profiles.

Key Features

• Keyword Search: Search npm with any keyword. Supports npm's special search syntax (e.g., keywords:front-end).
• Package Details: Scrape structured, detailed data for any package, including versions, maintainers, and historical metadata.
• User Packages: Retrieve all packages published by a specific user.
• Performance: Optimized for speed with efficient request handling.

How to Use

Configure the actor using a JSON input object. The scraper runs on Apify and stores its results in a dataset for later export via the Apify API.

Input Parameters

Provide input as a JSON object with the following fields:

• search: (Optional, String) A keyword to search for on npmjs.com.
• startUrls: (Optional, Array) A list of npm URLs to scrape. Accepts package detail, user profile, or search/list URLs.
• endPage: (Optional, Number) The final page number to scrape for search requests and paginated startUrls. Default is to scrape all pages.
• maxItems: (Optional, Number) Limit the total number of items scraped.
• proxy: (Required, Object) Proxy configuration. You must use a proxy, such as Apify Proxy.
• customMapFunction: (Optional, String) A JavaScript function (as a string) to transform each scraped object.

Tip: To scrape a specific list or page range, use startUrls. For example, providing a link to page 5 and setting "endPage": 6 will scrape only pages 5 and 6.

Input Example

{
  "startUrls": [
    "https://www.npmjs.com/package/lodash",
    "https://www.npmjs.com/~ljharb",
    "https://www.npmjs.com/search?q=keywords:front-end"
  ],
  "endPage": 5,
  "maxItems": 100,
  "proxy": {
    "useApifyProxy": true
  }
}

Output

During the run, the actor outputs progress logs. Results are stored in the run's dataset. Each item is a structured JSON object. For package details, this typically includes:
* url
* name
* description
* versions
* maintainers
* Other package metadata

You can access results using the Apify API in any language. See the Apify API reference for details.

Notes

• A proxy is required for operation.
• Compute unit consumption is typically low (~0.01-0.03 units for 100 listings).
• For bug reports or feature requests, create an issue on the GitHub repository.