Video-to-Playwright Automation Actor Transform screen recordings into executable Playwright automation scripts using Google Gemini AI vision capabilities. ## 🎯 What It Does This actor analyzes video recordings of user interactions with websites and automatically generates Python Playwright scripts that replicate those actions. Perfect for: - QA Testing: Record manual tests once, replay them automatically - Web Scraping: Show the actor how to navigate a site, get a working script - RPA Automation: Convert manual workflows into automated browser tasks - Documentation: Generate script documentation from video demos ## 🚀 How It Works 1. Upload Video: Place your screen recording (mp4, avi, mov, webm) in the videos directory 2. AI Analysis: Google Gemini 2.0 Flash analyzes every frame to identify: - Mouse clicks and movements - Keyboard inputs and text entries - Navigation patterns - Element interactions (buttons, forms, links) 3. Script Generation: Creates a complete, production-ready Playwright Python script with: - Proper selectors (CSS, text-based, IDs) - Wait conditions and timeouts - Error handling and screenshots on failure - Detailed comments explaining each step 4. Auto-Execution: Optionally runs the generated script to verify it works 5. Storage: Saves scripts to Apify key-value store for later use ## 📋 Input Parameters | Parameter | Type | Description | Default | |-----------|------|-------------|---------| | videoFile | string | Name of video file in videos directory | test_1.mp4 | | taskDescription | string | Optional context about the task shown | - | | includeScreenshots | boolean | Add screenshot captures to script | false | | executeAfter | boolean | Auto-run script after generation | true | | headless | boolean | Run browser in headless mode | false | | slowMo | integer | Slow motion delay (ms) for debugging | 100 | ## 🔧 Environment Variables Set these in the Apify Console: - GEMINI_API_KEY (required): Your Google Gemini API key - Get one here - VIDEO_UPLOAD_DIR: Custom directory for videos (default: ./videos) - MAX_VIDEO_SIZE_MB: Max upload size in MB (default: 100) - GEMINI_MODEL: Model to use (default: gemini-2.0-flash-exp) ## 📦 Outputs ### Key-Value Store - generated_script.py: The complete Playwright automation script - Additional scripts if you request modifications ### Dataset (if save_output=true) - Execution results with stdout/stderr - Success/failure status - Return codes ## 💡 Example Usage ### Basic Usage json { "videoFile": "login_workflow.mp4", "taskDescription": "User logging into dashboard", "executeAfter": true } ### Advanced Usage json { "videoFile": "checkout_process.mp4", "taskDescription": "Complete e-commerce checkout flow", "includeScreenshots": true, "executeAfter": true, "headless": false, "slowMo": 500 } ## 🎥 Video Recording Tips For best results when recording videos: 1. Clear Actions: Perform actions deliberately with visible mouse movements 2. Wait for Loads: Pause after page loads and before clicking 3. Stable Elements: Interact with elements that have consistent selectors 4. Resolution: Record in 1080p or higher for better element detection 5. Duration: Keep videos under 2-3 minutes (Gemini context limits) 6. Single Task: One clear workflow per video ## 🏗️ Architecture ┌─────────────────┐ │ Upload Video │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Gemini Vision │ ◄── Frame-by-frame analysis │ AI Analysis │ Interaction detection └────────┬────────┘ Element identification │ ▼ ┌─────────────────┐ │ Script Generator│ ◄── Playwright template │ (AI Prompted) │ Selector optimization └────────┬────────┘ Error handling │ ▼ ┌─────────────────┐ │ Execute Script │ ◄── Browser automation │ (Optional) │ Verification └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Save to Store │ ◄── Key-value storage │ Return Results│ Dataset output └─────────────────┘ ## 🛠️ Technical Details ### Supported Websites - Works best with public websites (no login walls for search engines) - Handles dynamic content (SPAs, AJAX) - Supports consent dialogs and popups - Compatible with YouTube, Google, e-commerce sites, etc. ### Script Features Generated scripts include: - Async/await pattern for modern Playwright - Proper browser context setup - Viewport and user-agent configuration - Timeout management (15s elements, 30s navigation) - Try/catch error handling - Screenshot on failure - Console logging for debugging ### Limitations - Video must be clear and not too fast - Complex multi-step workflows may need script refinement - Personalized content (like YouTube homepage) requires search functionality - Very long videos (>5 min) may hit token limits ## 🔄 MCP Server Mode The actor can also run as an MCP (Model Context Protocol) server for interactive use: Set AUTO_ANALYZE_VIDEO=false to enable MCP mode, then use these tools: - analyze_video: Generate script from video - modify_script: Refine script with natural language - execute_script: Run the automation - get_script: Retrieve current script - save_script: Store to key-value store ## 📚 Example Output python from playwright.async_api import async_playwright import asyncio async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=False) page = await browser.new_page() # Step 1: Navigate to YouTube print("Navigating to YouTube...") await page.goto('https://www.youtube.com') # Step 2: Search for video print("Searching...") search_input = page.locator('input[name="search_query"]') await search_input.fill('Minecraft gameplay') await page.keyboard.press('Enter') await asyncio.sleep(2) # Step 3: Click first video print("Clicking video...") video = page.locator('a#video-title').nth(0) await video.click() print("[SUCCESS] Automation completed!") await browser.close() asyncio.run(main()) ## 🐛 Troubleshooting Script times out finding elements: - Video content may not be available (e.g., personalized feeds) - Use search functionality instead of expecting content on homepage - Increase timeout values in generated script Unicode/encoding errors: - Already handled with UTF-8 subprocess environment - If issues persist, remove emojis from print statements Video upload fails: - Check file size (max 100MB by default) - Ensure video format is supported (mp4, avi, mov, webm) - Verify video path in VIDEO_UPLOAD_DIR Generated script doesn't work: - Use modify_script tool to refine selectors - Add more wait conditions - Handle dynamic content with explicit waits ## 📄 License MIT ## 🔗 Links - Apify Platform - Playwright Documentation - Google Gemini API - MCP Protocol ## 👥 Support For issues or questions: 1. Check the execution logs in Apify Console 2. Review generated script for errors 3. Try modifying the script with natural language instructions 4. Ensure your Gemini API key is valid and has quota