Precious Metal Price Scraper A comprehensive web scraping actor for collecting precious metal prices from global and Indonesian sources with AI-powered trend analysis. ## Description This actor aggregates gold and metal prices from over 20 sources worldwide, including major global exchanges and Indonesian local markets. It provides real-time price data, historical trends, and AI-powered analysis for investment decision-making. The scraper covers multiple precious metals including gold, silver, platinum, and palladium, with automatic currency conversion and quality assessment. ## Features - Multi-source Data Collection: Aggregates prices from 20+ global and Indonesian sources - AI Trend Analysis: Provides intelligent trend analysis with momentum indicators and volatility scoring - Data Normalization: Converts all prices to standard units (oz, gram, kg) and currencies (USD, IDR) - Quality Assessment: Evaluates data quality with ratings (complete, partial, estimated, limited) - Error Handling: Robust error handling and logging for failed scraping attempts - Structured Output: Well-formatted JSON output compliant with defined schema - Real-time Processing: Near real-time price updates from multiple sources ## Data Sources ### Global Sources - GoldPrice.org - Kitco - TradingEconomics - Investing.com - GoldBroker - BullionVault - Metals Daily - LBMA (London Bullion Market Association) - Nasdaq ### Indonesian Local Sources - Logam Mulia (Antam) - Galeri24 - Pegadaian - Tokopedia - Indogold - Orori - Lakuemas - HargaEmas.org ### API Sources - GoldAPI.io - Metals-API.com ## Input The actor accepts JSON input with the following parameters: ```json { "metals": ["gold", "silver"], "sources": ["global", "local", "api"], "use_proxy": true } Input Parameters metals (array, optional): Types of metals to collect data for. Default: ["gold", "silver"] Options: "gold", "silver", "platinum", "palladium" sources (array, optional): Types of data sources to use. Default: ["global", "local"] Options: "global", "local", "api" use_proxy (boolean, optional): Whether to use proxy for scraping. Default: true Output The actor produces structured JSON data with the following format: json { "metalType": "gold", "source": "goldprice.org", "sourceUrl": "https://goldprice.org/", "priceBid": 1975.25, "priceAsk": 1976.75, "priceMid": 1976.0, "currency": "USD", "unit": "oz", "pricePerGram": 63.51, "pricePerKilo": 63510.0, "priceIDR": 30628000.0, "pricePerGramIDR": 984400.0, "pricePerKiloIDR": 984400000.0, "changePercent": 0.25, "changeValue": 4.92, "dayHigh": 1980.5, "dayLow": 1970.25, "yearHigh": 2080.0, "yearLow": 1810.5, "trend": "uptrend (moderate)", "trendReason": "Moderate upward movement with 0.25% increase", "volatilityScore": 0.35, "marketTime": "2023-11-29T17:45:00.000Z", "scrapedAt": "2023-11-29T17:45:30.000Z", "region": "Global", "market": "International", "exchange": "GoldPrice.org", "dataQuality": "complete", "status": "active" } Output Fields metalType: Type of metal (gold, silver, platinum, palladium) source: Name of the data source sourceUrl: Original source URL priceBid/priceAsk/priceMid: Bid/ask/mid prices currency/unit: Price currency and unit pricePerGram/pricePerKilo: Price per gram/kilo in original currency priceIDR/pricePerGramIDR/pricePerKiloIDR: Prices in Indonesian Rupiah changePercent/changeValue: Percentage and absolute price change dayHigh/dayLow/yearHigh/yearLow: Daily and yearly high/low prices trend/trendReason: AI trend analysis and reasoning volatilityScore: Volatility score (0-1 scale) marketTime/scrapedAt: Market and scraping timestamps region/market/exchange: Market and exchange information dataQuality: Data quality rating (complete, partial, estimated, limited) status: Data status (active, error) Installation Using Apify Console Create a new actor in Apify Console Select the "Python Crawlee with Playwright" template Replace the main.py code with this project's code Upload the dataset_schema.json file to define the output schema Local Development Install Apify CLI: bash Line Wrapping # Using Homebrew brew install apify-cli # Or using NPM npm -g install apify-cli Pull the actor from Apify Console: bash Line Wrapping Collapse 1 apify pull Run locally: bash Line Wrapping Collapse 1 apify run Usage Via Apify Console Navigate to the actor page in Apify Console Click on the "Input" tab Enter your desired input parameters Click "Run" to start the scraping process View results in the "Dataset" tab Via API bash Line Wrapping curl -X POST https://api.apify.com/v2/acts//runs?token= \ -d '{ "metals": ["gold", "silver"], "sources": ["global", "local"], "use_proxy": true }' Example Use Cases Investment Research: Collect comprehensive price data for precious metals investment analysis Market Monitoring: Monitor real-time price movements across multiple exchanges Price Comparison: Compare prices between global markets and Indonesian local markets Trend Analysis: Leverage AI-powered trend analysis for trading decisions Data Integration: Feed structured data into other financial systems or dashboards Data Views The actor provides multiple pre-configured dataset views: Live Prices: Real-time prices across all sources Best Prices: Find the lowest prices for each metal Global Markets: International spot prices (USD/oz) Indonesia Market: Local Indonesian prices (IDR/gram) Price Movers: Biggest price movements sorted by percentage change By Source: Compare prices from all sources Gold Only: Gold prices from all sources Trend Analysis: AI-powered trend analysis with volatility scores Detailed View: Complete data with all fields Errors: Failed scraping attempts Error Handling The actor includes comprehensive error handling: Failed scraping attempts are logged with detailed error messages Error responses are stored in the dataset with consistent structure Data validation ensures only properly formatted data is stored Retry mechanisms for temporary failures Rate Limiting To avoid overwhelming target websites: Requests are distributed across multiple proxy servers Delays are implemented between requests to the same domain Concurrent requests are limited to prevent server overload Contributing This project is maintained by Supernesia Creative Technology. For contributions or issues, please contact the development team. License This project is proprietary software of Supernesia Creative Technology. All rights reserved. Support For support and inquiries: Email: info@supernesia.id Website: https://supernesia.id