LinkedIn Agent

A production-ready LinkedIn scraping agent built with clean architecture principles. It's structured for maintainability and scalability, featuring a Python backend, React dashboard, and full Docker support.

Overview

This actor scrapes LinkedIn profiles and company pages. It handles batch processing through Excel/CSV files or direct JSON input, manages jobs via a Redis queue, and provides a web dashboard for monitoring. The system is designed to run both locally and on the Apify platform.

Key Features

• Clean Architecture: Separated backend (Python), frontend (React), and infrastructure layers.
• Queue Processing: Redis-based job queue with worker processes for reliable execution.
• Batch Processing: Accepts Excel/CSV files for bulk operations with resumable jobs.
• Modern Dashboard: React-based admin UI to monitor and manage scraping jobs.
• Docker-First: Multi-stage Docker builds for development and production.
• Security: JWT authentication and role-based access control.
• Monitoring: Built-in health checks and Prometheus integration.
• Persistent Storage: SQLite database with automated backups.

How to Use

Quick Start (Development)

git clone <your-repo>
cd linkedin-agent
make setup-dev
# Edit .env file with your API keys
make dev

Production Deployment

make setup-prod
# Configure .env for production
make deploy

Common Commands

make help          # List all commands
make dev           # Start backend services
make fullstack-dev # Start backend + frontend
make deploy        # Deploy to production
make status        # Check service status
make logs          # View logs

API Usage

Submit a job via REST API:

curl -F "owner_email=user@example.com" \
     -F "input_file=@examples/sample_input.xlsx" \
     http://localhost:8000/api/jobs/submit

Input / Output

Input Formats

The agent accepts three input types:

1. JSON: Direct LinkedIn URLs in a structured JSON object.
2. Excel/CSV: Batch files for processing multiple profiles.
3. REST API: Programmatic submission as shown above.

Excel Template

Use the provided templates (examples/linkedin_template.xlsx or examples/sample_input.xlsx). Files must contain these sheets:
* Company_Profiles - For LinkedIn company page URLs
* Individual_Profiles - For LinkedIn personal profile URLs

Configuration

Copy and edit the environment file:

cp examples/env.example .env

Key variables to set:
* OPENAI_API_KEY - For LLM features
* APIFY_TOKEN - For Apify platform and proxy features
* SUPABASE_JWT_SECRET - For authentication

Output

Scraped data is stored in the SQLite database and can be exported via the admin dashboard. The system maintains job history, status, and error logs for all processed requests.

Documentation

Full documentation is available in the docs/ directory:
* PROJECT_STRUCTURE.md - Detailed folder organization
* docs/ARCHITECTURE.md - System design and components
* docs/API.md - Complete REST API reference
* docs/DEPLOYMENT.md - Production deployment guide
* docs/EXCEL_FORMAT.md - Detailed input format specifications