Web Link Graph Visualizer Creates oriented graphs visualizing links between webpages Crawl a website starting from a URL, extract all links, build a directed graph of the link structure, and export it as a PNG image or JSON file. --- ## 📥 What You'll Get After the actor completes, you'll receive: ### 🖼️ graph.png - Visual Network Diagram - Location: Key-Value Store → graph.png - Format: High-resolution PNG (2000x1600px) - Content: Visual graph with color-coded nodes and directed edges - Download: Click "Actions" → "Download" in Key-Value Store tab ### 📊 graph.json - Structured Data - Location: Key-Value Store → graph.json - Format: JSON file with complete graph structure - Content: All nodes, edges, and statistics - Use: Import into analysis tools or custom visualizations ### 📑 Dataset - All Crawled Pages - Location: Dataset tab (Storage section) - Format: JSON records (one per page) - Content: URL, title, depth, all links per page - Export: CSV, JSON, or Excel from Dataset tab ### 🔍 Where to Find in Apify Console: 1. After actor finishes, go to "Storage" section 2. Key-Value Store tab: - Download graph.png (your visual graph image) - Download graph.json (data for analysis) 3. Dataset tab: - View/export all crawled pages - See links extracted from each page --- ## Features ✅ Smart Crawling: - Start from any URL - Follow links matching a boundary regex - Configurable depth and page limits - Respects robots.txt (via Playwright) - Adjustable request delays ✅ Graph Building: - Directed graph (oriented edges) - Track internal vs external links - URL normalization (remove fragments, trailing slashes) - Depth tracking for each node - Duplicate link detection ✅ Visualization: - Multiple layout algorithms (hierarchical, spring, circular, random) - Customizable node labels (URL, path, title, or index) - Color-coded nodes (internal=blue, external=red) - High-resolution PNG export - JSON export for programmatic use ✅ Statistics: - Total nodes and edges - Average outgoing links per page - Max depth reached - Internal vs external link counts --- ## Input Parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | startUrl | String | Required | The URL to start crawling from | | boundaryRegex | String | .* | Regex to limit which URLs to crawl | | maxDepth | Integer | 3 | Maximum crawl depth (1-10) | | maxPages | Integer | 50 | Maximum pages to crawl (1-1000) | | exportFormat | Select | both | Output format: both, image, or json | | graphLayout | Select | hierarchical | Layout: hierarchical, spring, circular, random | | nodeLabels | Select | path | Label type: url, path, title, index | | includeExternal | Boolean | true | Show external links in graph | | waitForSelector | String | - | CSS selector to wait for (optional) | | requestDelay | Integer | 1000 | Delay between requests (ms) | --- ## Example Inputs ### Example 1: Small Website json { "startUrl": "https://example.com", "boundaryRegex": "^https://example\\.com/.*", "maxDepth": 2, "maxPages": 20, "exportFormat": "both", "graphLayout": "hierarchical", "nodeLabels": "path" } ### Example 2: Documentation Site json { "startUrl": "https://docs.python.org/3/", "boundaryRegex": "^https://docs\\.python\\.org/3/tutorial/.*", "maxDepth": 3, "maxPages": 50, "exportFormat": "image", "graphLayout": "spring", "nodeLabels": "title", "includeExternal": false, "requestDelay": 500 } ### Example 3: Blog with Subdomains json { "startUrl": "https://blog.example.com", "boundaryRegex": "^https://.*\\.example\\.com/.*", "maxDepth": 2, "maxPages": 30, "exportFormat": "both", "graphLayout": "circular", "nodeLabels": "path" } --- ## Output ### Dataset Each crawled page is saved to the dataset with: - url - Page URL - title - Page title - depth - Depth from start URL - links - All extracted links - internal_links - Links matching boundary - external_links - Links outside boundary - crawled_at - Timestamp ### Key-Value Store graph.json (if JSON export enabled): json { "graph": { "nodes": [ { "id": "https://example.com", "url": "https://example.com", "title": "Example Domain", "depth": 0, "is_internal": true, "outgoing_links": 3 } ], "edges": [ { "source": "https://example.com", "target": "https://example.com/page1" } ], "directed": true }, "statistics": { "nodes": 15, "edges": 42, "crawled_pages": 15, "external_links": 3, "avg_outgoing_links": 2.8, "max_depth_reached": 2 } } graph.png (if image export enabled): - High-resolution PNG image (2000x1600px) - Color-coded nodes (blue=internal, red=external) - Directed edges with arrows - Legend and statistics OUTPUT: json { "start_url": "https://example.com", "statistics": { "nodes": 15, "edges": 42, "crawled_pages": 15 }, "exports": { "json": true, "image": true } } --- ## Boundary Regex Examples | Pattern | Matches | |---------|---------| | ^https://example\\.com/.* | All pages on example.com | | ^https://example\\.com/blog/.* | Only blog section | | ^https://.*\\.example\\.com/.* | All subdomains | | ^https://example\\.com/(?!admin).* | Exclude admin section | | .* | Everything (no boundary) | --- ## Use Cases 🔍 SEO Analysis: - Visualize site structure - Find orphan pages - Identify link depth issues 📊 Content Strategy: - Map content relationships - Find hub pages - Identify external dependencies 🔗 Link Building: - Discover internal linking opportunities - Find broken link paths - Analyze link distribution 🛠️ Site Migration: - Document current structure - Plan URL redirects - Validate link integrity --- ## Graph Layouts ### Hierarchical (Default) Best for: Sites with clear hierarchy (docs, blogs) - Top-down structure - Shows depth clearly ### Spring (Force-Directed) Best for: Discovering clusters - Nodes repel/attract based on connections - Reveals natural groupings ### Circular Best for: Small sites - Nodes arranged in a circle - Shows connections clearly ### Random Best for: Quick visualization - Fast to generate - Good for dense graphs --- ## Node Label Types | Type | Example | Best For | |------|---------|----------| | url | https://example.com/page | Small graphs | | path | /blog/post-title | Medium graphs (default) | | title | My Blog Post | Readable labels | | index | 1, 2, 3 | Large graphs | --- ## Performance Tips 1. Start Small: - Use maxPages: 20 for initial runs - Increase gradually 2. Tight Boundaries: - Use specific regex patterns - Avoid crawling entire domains 3. Adjust Depth: - Depth 2-3 is usually sufficient - Depth 4+ can explode exponentially 4. Request Delays: - Use 1000ms+ for courtesy - Reduce for fast sites 5. External Links: - Set includeExternal: false for cleaner graphs - Enable to see dependencies --- ## Limitations - Max Pages: 1000 (configurable limit) - Max Depth: 10 (configurable limit) - JavaScript: Rendered via Playwright (may be slow) - Image Size: Large graphs (100+ nodes) may have small labels --- ## Technical Details Built With: - Python 3.11 - Apify SDK - Playwright (browser automation) - BeautifulSoup4 (HTML parsing) - NetworkX (graph algorithms) - Matplotlib (visualization) Graph Type: - Directed graph (DiGraph) - Nodes = URLs - Edges = Links (from → to) URL Normalization: - Removes fragments (#section) - Removes trailing slashes - Preserves query strings - Converts relative to absolute --- ## Example Output ### Small Site (10 pages) Nodes: 10 Edges: 28 Crawled pages: 10 External links: 3 Avg links per page: 2.8 Max depth reached: 2 ### Documentation Site (50 pages) Nodes: 53 (50 internal + 3 external) Edges: 142 Crawled pages: 50 External links: 3 Avg links per page: 2.7 Max depth reached: 3 --- ## Troubleshooting Issue: No links found - Check waitForSelector for dynamic sites - Verify boundary regex matches start URL Issue: Too many nodes - Reduce maxPages or maxDepth - Tighten boundary regex Issue: Image labels too small - Use nodeLabels: "index" for large graphs - Reduce number of nodes Issue: Slow crawling - Reduce requestDelay - Decrease maxPages - Check site performance --- ## Support For issues or questions: 1. Check input parameters 2. Verify boundary regex 3. Test with small maxPages first 4. Review dataset for crawl results --- ## License MIT License - Free for commercial and personal use --- Built with ❤️ using Apify SDK