AI Training Data Enricher & Validator

Overview

An Apify actor for cleaning, enriching, and validating datasets before LLM fine-tuning. It processes your data to remove duplicates, detect sensitive information, and add analytical metadata, helping to prevent overfitting, privacy violations, and bias in trained models.

Key Features

Enrichment
* Sentiment Analysis: Scores text from -5 (negative) to +5 (positive) using the AFINN lexicon.
* Named Entity Recognition: Extracts people, places, organizations, dates, and values.
* Keyword Extraction: Identifies key terms using TF-IDF weighting.
* Language Detection: Detects text language with a confidence score.
* Readability Metrics: Calculates word count, sentence count, and complexity scores.

Validation
* Duplicate Detection: Fuzzy string matching with a configurable similarity threshold (0.5 to 1.0).
* PII Detection: Finds emails, phone numbers, SSNs, and credit card numbers for GDPR compliance.
* Schema Validation: Validates items against a JSON Schema.
* Length Filtering: Enforces minimum and maximum character limits.
* Quality Flags: Optional "flag-only" mode to mark invalid items without removing them.

Privacy & Compliance
* PII Redaction: Can automatically replace detected sensitive data with [REDACTED].
* Audit Trail: Provides a complete validation history for each item.

How to Use

Input Data Format

Prepare a dataset where each item contains at least a text field.

{
  "text": "Your training sample text here.",
  "label": "optional_label"
}

Basic Configuration

Run the actor with a configuration like this:

{
  "datasetId": "your-dataset-id",
  "textField": "text",
  "enrichmentOptions": {
    "sentiment": true,
    "entities": true,
    "keywords": true,
    "language": true,
    "readability": true
  },
  "validationOptions": {
    "detectDuplicates": true,
    "duplicateSimilarityThreshold": 0.85,
    "detectPII": true,
    "minTextLength": 10,
    "maxTextLength": 0
  },
  "outputOptions": {
    "includeOriginal": true,
    "flagOnly": false,
    "removePII": false
  }
}

Input / Output

Input: Your source dataset via an Apify dataset ID.

Output: A new dataset with enriched and validated items. Each output item has the following structure:

{
  "id": 0,
  "originalText": "Apple Inc. released iPhone in 2007. Great product!",
  "enrichment": {
    "sentiment": {
      "score": 3,
      "comparative": 0.375,
      "positive": ["great"],
      "negative": []
    },
    "entities": {
      "organizations": ["Apple Inc."],
      "dates": ["2007"]
    },
    "keywords": ["apple", "iphone", "released", "product"],
    "language": "english",
    "readability": {
      "wordCount": 8,
      "sentenceCount": 2
    }
  },
  "validation": {
    "isValid": true,
    "isDuplicate": false,
    "hasPII": false,
    "lengthValid": true,
    "schemaValid": true
  }
}