Dataset Quality Scorer

An Apify actor that automatically scores and diagnoses the quality of machine learning datasets. It checks for common data issues, detects drift and outliers, and generates actionable reports to improve data health before model training.

Key Features

• Quality Scoring: Calculates a comprehensive score based on completeness, consistency, duplicates, and class balance.
• Data Drift Detection: Compares a current dataset against a baseline to identify significant distribution shifts.
• Outlier Detection: Finds anomalies using multiple methods (Z-score, IQR, Isolation Forest).
• Schema Validation: Ensures data matches an expected structure, checking column presence and data types.
• Full Report Generation: Runs all checks and combines results into a single, detailed analysis.

How to Use

Choose an operation and provide your dataset via datasetUrl (a URL or Apify dataset ID) or inline as datasetData. Configure the specific checks you need.

Common Operations & Parameters

• scoreQuality: Scores the dataset. Use checkDuplicates and checkBalance flags; specify targetColumn for balance analysis.
• detectDrift: Compares against a baselineDataset to find distribution changes.
• findOutliers: Detects anomalies. Choose an outlierMethod (zscore, iqr, isolation_forest) and set the outlierThreshold.
• validateSchema: Validates data structure against a provided schemaDefinition.
• generateReport: Runs a full suite analysis (recommended).

Example Input

To run a full report with schema validation and outlier detection:

{
  "operation": "generateReport",
  "datasetUrl": "https://example.com/data.csv",
  "schemaDefinition": {
    "columns": {
      "id": "number",
      "feature1": "number",
      "label": "string"
    },
    "required": ["id", "label"]
  },
  "checkDuplicates": true,
  "checkBalance": true,
  "targetColumn": "label",
  "outlierMethod": "zscore",
  "outlierThreshold": 3
}

Input/Output

Main Input Parameters:
* operation (required): The analysis to run (scoreQuality, detectDrift, findOutliers, validateSchema, generateReport).
* datasetUrl / datasetData: The dataset to analyze.
* schemaDefinition: JSON schema for validation.
* baselineDataset: Baseline dataset for drift detection.
* outlierMethod: Algorithm for outlier detection.
* outlierThreshold: Sensitivity threshold for outliers (default: 3).
* checkDuplicates, checkBalance: Toggle specific quality checks.
* targetColumn: Column name for class balance analysis.

Output:
The actor returns a structured JSON object containing quality scores, lists of detected issues (like missing values, duplicates, drift, or outliers), and specific recommendations for improving the dataset.