Kalshi Market Data Scraper This Actor scrapes comprehensive market data from the Kalshi API including: - Events (prediction market categories) - Markets (individual betting markets with enhanced price analytics) - Orderbooks (current bid/ask prices and liquidity depth) - Recent trades (for historical price calculations) ## Enhanced Price Analytics The scraper now calculates comprehensive price metrics to match professional trading datasets: ### Core Price Data - last_price - Current market price - yes_bid, yes_ask, no_bid, no_ask - Current orderbook prices - bid_ask_spread - Difference between best bid and ask ### Historical Price Analysis - prev_day_price - Price 24 hours ago - prev_hour_price - Price 1 hour ago - prev_week_price - Price 7 days ago - day_change - Absolute price change over 24 hours - hour_change - Absolute price change over 1 hour - day_change_pct - Percentage price change over 24 hours - hour_change_pct - Percentage price change over 1 hour ### Volume & Liquidity Metrics - volume - Total contract volume - volume_24h - Trading volume in last 24 hours - recent_volume - Trading volume in last hour - dollar_volume - Total volume in dollar terms - dollar_recent_volume - Recent volume in dollar terms - dollar_open_interest - Open interest in dollar terms - open_interest - Total open contracts - liquidity - Total orderbook depth ### Market Mechanics - settlement_timer_seconds - Time until settlement - can_close_early - Whether market can close before expiration - close_unconfirmed - Early closure status - risk_limit_cents - Position risk limits - min_tick_size - Minimum price increment - mutually_exclusive - Whether market is mutually exclusive - settle_details - Settlement mechanism details - description_context - Additional market context ## Configuration ### Input Parameters json { "environment": "production", "includeEvents": true, "includeMarkets": true, "includeOrderbooks": true, "includeTrades": true, "maxEvents": 0, "maxMarketsPerEvent": 0, "enhancedPriceMetrics": true, "eventStatusFilter": "open" } - environment: "demo" or "production" - includeEvents: Scrape event data (default: true) - includeMarkets: Scrape market data (default: true) - includeOrderbooks: Scrape orderbook data for price calculations (default: true) - includeTrades: Scrape trade history for price analytics (default: true) - maxEvents: Limit number of events (0 = no limit) - maxMarketsPerEvent: Limit markets per event (0 = no limit) - eventStatusFilter: Filter events by status - "open" (default), "closed", "settled", or "all" ### Event Status Filter Options - "open" (default): Only collect active events where trading is still happening - "closed": Only collect events where trading has ended but results aren't final - "settled": Only collect events with final results determined - "all": Collect all events regardless of status The default "open" filter is recommended for most use cases as it focuses on actively tradeable markets. ### Credentials Set up your Kalshi API credentials in Apify secure storage: - KALSHI_API_KEY: Your Kalshi Key ID - KALSHI_PRIVATE_KEY: Your RSA private key in PEM format ## Output Format The scraper outputs comprehensive JSON data that can be converted to CSV with all price analytics: json { "type": "market", "scraped_at": "2025-01-17T10:30:00Z", "market_info": { "ticker": "EXAMPLE-24Q1", "last_price": 0.52, "yes_bid": 0.51, "yes_ask": 0.53, "bid_ask_spread": 0.02, "prev_day_price": 0.48, "day_change": 0.04, "day_change_pct": 8.33, "volume_24h": 1250, "dollar_volume": 650, "liquidity": 5000, "open_interest": 2500, "can_close_early": true, "settlement_timer_seconds": 86400, // ... additional comprehensive fields }, "orderbook": { "yes": [...], "no": [...] }, "recent_trades": [...] } ## Comprehensive Data Collection This enhanced version collects the same comprehensive data as professional trading platforms: 1. Real-time pricing with bid/ask spreads 2. Historical price analysis with change calculations 3. Volume analytics across multiple timeframes 4. Liquidity metrics from orderbook depth 5. Market mechanics and settlement details 6. Trade history for price discovery analysis The output matches the structure of professional Kalshi datasets with 40+ data fields per market. ## Usage For comprehensive data collection, use the included input_comprehensive.json configuration which enables all enhanced features. The scraper automatically calculates derived metrics from orderbook and trade data, providing the complete picture needed for market analysis and trading strategies. ## Features - Comprehensive Data Collection: Events, markets, orderbooks, and recent trades - Secure Credential Management: Uses Apify secure storage for API credentials - Environment Support: Works with both Kalshi demo and production environments - Robust Error Handling: Automatic retries with exponential backoff - Rate Limiting: Smart delays to respect API limits - Statistics Tracking: Detailed progress and performance metrics - Flexible Configuration: Customizable data collection options ## Setup ### 1. Kalshi API Credentials Get your API credentials from Kalshi Account Profile: 1. Go to the API Keys section 2. Create a new API key if you don't have one 3. Note down your Key ID and download the Private Key file 4. The private key should be in PEM format starting with -----BEGIN RSA PRIVATE KEY----- ### 2. Secure Storage Setup (Recommended) For security, store your credentials in Apify's secure storage instead of hardcoding them: #### Option A: Use the Helper Script 1. Put your credentials in config.env: bash KALSHI_API_KEY=your-api-key-id KALSHI_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY----- your-private-key-content-here -----END RSA PRIVATE KEY-----" 2. Run the setup script: bash python3 setup_secure_storage.py #### Option B: Manual Setup In your Apify console or via API, set these key-value pairs: - KALSHI_API_KEY: Your Kalshi API Key ID - KALSHI_PRIVATE_KEY: Your RSA private key in PEM format ### 3. Local Development Setup For local testing and development: 1. Clone this repository 2. Install dependencies: bash pip install -r requirements.txt 3. Set up your credentials in config.env: bash # Copy the template cp config.env.template config.env # Edit with your credentials nano config.env 4. Run locally: bash python3 load_from_env.py ## Performance Notes - Events + Markets only: ~2-5 minutes for 400+ events - With Orderbooks: Significantly longer due to API rate limits - With Trades: Additional time depending on market activity - Rate Limiting: Built-in delays prevent API throttling ## Troubleshooting ### Authentication Issues ❌ Authentication failed: Invalid signature - Verify your private key is in correct PEM format - Ensure no extra whitespace or formatting issues - Check that you're using the correct API Key ID ### API Rate Limits ⚠️ Rate limit encountered, waiting... - The scraper automatically handles rate limits - Consider reducing maxEvents or maxMarketsPerEvent for faster runs - Disable orderbooks/trades for much faster scraping ### Local Development Issues ❌ No credentials found in any source! - Make sure config.env file exists with proper credentials - Verify the file format matches the template - Check that environment variables are set correctly ### Missing Data ⚠️ Failed to fetch orderbook for market XYZ - Some markets may not have orderbook data available - Network timeouts are automatically retried - Check market status (some may be closed) ## API Documentation For more details about the Kalshi API: - Kalshi API Documentation - Trading API Reference ## Support If you encounter issues: 1. Check the troubleshooting section above 2. Verify your API credentials and permissions 3. Review the Apify Actor logs for detailed error messages 4. Ensure you're using the correct environment (demo vs production) ## License This project is open source. See the Apify platform terms for usage guidelines. ## Changelog ### v0.2.0 - Added retry logic with exponential backoff - Enhanced error handling and recovery - Added runtime statistics tracking - Improved rate limiting - Added environment variable support - Enhanced logging and progress reporting ### v0.1.0 - Initial release - Support for events, markets, orderbooks, and trades - RSA-based authentication - Demo and production environment support - Configurable data collection parameters