Job Listings Aggregator

A Python application that crawls multiple public job boards, normalizes the listings, and provides a unified interface for searching and managing them. It's designed for automation and can be scheduled to run daily.

Key Features

• Multi-source scraping: Fetches jobs from sites like RemoteOK and We Work Remotely. The modular architecture makes it easy to add new job boards.
• Data processing: Automatically normalizes and deduplicates job listings from different sources into a consistent format.
• Flexible storage: Saves aggregated jobs to either JSON files or a SQLite database.
• Advanced search & filtering: Filter jobs by keywords, location (including "remote"), job type, and company.
• Export results: Export filtered job lists to CSV or JSON files.
• CLI & API: Use via a command-line interface or import directly into your Python code.
• Automated scheduling: Set up daily, automated scraping runs.

How to Use

Installation & Setup

1. Clone the repository and install dependencies:
   bash git clone <repository-url> cd job-listings-aggregator pip install -r requirements.txt
2. (Optional) Install Playwright browsers for JavaScript-heavy sites:
   bash playwright install
3. Configure the app by editing config.py. Key settings include STORAGE_TYPE ('json' or 'sqlite'), MAX_JOBS_PER_BOARD, and scheduler options.

Basic Commands (CLI)

Scrape Jobs:

# Scrape all enabled job boards
python main.py scrape

# Scrape with specific keywords
python main.py scrape -k python -k remote

# Limit the number of jobs per board
python main.py scrape -m 50

Search & Filter Jobs:

# Search by keyword
python main.py search -k python

# Filter for remote jobs
python main.py search --remote

# Complex filter (keyword, location, company)
python main.py search -k "data scientist" -l remote --company google

# Export results
python main.py search -k python --export results.csv

Manage Automation:

# Start/stop the daily scheduler
python main.py schedule --start
python main.py schedule --stop

# Run a scraping job immediately
python main.py schedule --run-now

Python API

You can also use the aggregator directly in your scripts:

from main import JobAggregator

app = JobAggregator()
jobs = app.scrape_all_jobs(keywords=['python', 'remote'])
app.save_jobs(jobs)

remote_jobs = app.search_jobs(remote_only=True, keyword='developer')
for job in remote_jobs[:5]:
    print(f"{job.title} at {job.company}")

Input / Output

• Input: The tool takes its input from configured job board websites. You control the process via CLI commands (keywords, limits) or the configuration file (storage type, scheduling).
• Processing: It scrapes, normalizes fields (title, company, location, link), and deduplicates listings.
• Output: Jobs are saved to your chosen storage backend (JSON file or SQLite DB). Search results can be viewed in the terminal or exported to CSV/JSON files.