Tenders Direct Notice Scraper Apify Actor that scrapes tender notice details from Tenders Direct. The Actor handles authentication with persistent session storage and extracts notice details from individual notice pages. ## Features - Robust Authentication: Handles login with session persistence across runs - Azure AD Support: Works with both standard login forms and Azure AD SSO flows - Configurable Timeouts: Adjust navigation timeouts for different network conditions - Debug Mode: Toggle browser visibility and debug screenshots for troubleshooting - Flexible Input: Accepts single URLs or arrays of URLs in multiple formats ## Input Configuration The Actor accepts the following input fields (all optional except notice URLs): ### Required - noticeURL (array or string) - One or more Tenders Direct notice URLs to scrape - Accepts: single string, array of strings, or array of {url: "..."} objects - Example: ["https://app.tendersdirect.co.uk/notice/12345"] ### Authentication - email (string) - Login email for Tenders Direct - Falls back to TD_EMAIL environment variable if not provided - Example: "your.email@company.com" - password (string, secret) - Login password for Tenders Direct - Falls back to TD_PASSWORD environment variable if not provided - Stored securely and masked in UI ### Timeouts - loginTimeout (integer, default: 25000) - Maximum time in milliseconds to wait for login page navigation - Increase for slow networks or Azure AD SSO flows - Range: 5000-120000ms - noticeTimeout (integer, default: 30000) - Maximum time in milliseconds to wait for notice page navigation - Increase for large or slow-loading notice pages - Range: 5000-120000ms ### Debug Options - headless (boolean, default: true) - Run browser in headless mode (no visible window) - Set to false when debugging login issues locally - saveDebugScreenshots (boolean, default: true) - Save screenshots and HTML to Key-Value Store when errors occur - Useful for debugging but consumes storage - Set to false to reduce storage usage in production ## Example Input ### Minimal (using environment variables) json { "noticeURL": [ "https://app.tendersdirect.co.uk/notice/12345", "https://app.tendersdirect.co.uk/notice/67890" ] } Note: Set TD_EMAIL and TD_PASSWORD in Actor environment variables. ### Complete (all options) json { "noticeURL": [ "https://app.tendersdirect.co.uk/notice/12345", "https://app.tendersdirect.co.uk/notice/67890" ], "email": "your.email@company.com", "password": "your-password", "loginTimeout": 45000, "noticeTimeout": 60000, "headless": true, "saveDebugScreenshots": true } ## Output The Actor pushes data to the default dataset with the following structure: json { "url": "https://app.tendersdirect.co.uk/notice/12345", "title": "Notice Title", "noticeDetailsHtml": "<div>...</div>", "noticeDetailsText": "Plain text content...", "scrapedAt": "2025-01-15T10:30:00.000Z" } ## Local Development ### Prerequisites bash npm install ### Environment Setup Create environment variables or provide them via input: bash export TD_EMAIL="your.email@company.com" export TD_PASSWORD="your-password" ### Run Locally bash npm start ### Code Quality bash # Format code npm run format # Check formatting npm run format:check # Lint code npm run lint # Fix lint issues npm run lint:fix ## Troubleshooting ### Login Failures 1. Set headless: false to see the browser and debug visually 2. Check debug screenshots in Key-Value Store (login-fail.html, login-fail.png) 3. Increase loginTimeout if using Azure AD SSO (try 45000-60000ms) 4. Verify credentials are correct ### Notice Extraction Failures 1. Check debug screenshots (notice-miss.html, notice-miss.png) 2. Increase noticeTimeout for slow-loading pages 3. Verify the notice URL is correct and accessible when logged in ### Session Issues - The Actor saves session state to avoid repeated logins - If experiencing auth issues, the session state is automatically refreshed on login - Session state is stored in Actor's Key-Value Store as storageState.json ## Migration from Environment Variables If you're currently using TD_EMAIL and TD_PASSWORD environment variables: 1. Your Actor will continue working without changes (backward compatible) 2. To migrate, add email and password to your input JSON 3. Input fields take precedence over environment variables 4. You can remove environment variables after migration ## Technical Details - Built with Crawlee and Playwright - Uses Chromium browser - Docker base image: apify/actor-node-playwright-chrome:22-1.54.1 - Node.js with ES modules ## Resources - Apify Platform documentation - Crawlee documentation - Playwright documentation - Apify SDK for JavaScript ## Support For issues or questions: - Check debug screenshots in Key-Value Store - Review Actor logs for error details - Adjust timeout values for your network conditions