> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Automated Retraining

> Automated weekly retraining DAG with 4 triggers: time-based, performance, data volume, and drift detection

OperationsData Science

> [!NOTE]
>
> **What this does:** The system continuously monitors prediction quality and retrains itself weekly when it detects degradation — ensuring forecasts stay accurate as real-world conditions change, without human intervention.

## Overview

Models degrade over time. The system employs an automated **Airflow DAG** (`dags/automated_retraining_dag.py`) that runs weekly and monitors 4 distinct triggers to decide when to launch a new training job.

> [!NOTE]
>
> The retraining pipeline runs as a `@taskflow` decorated Airflow DAG, not a standalone Python class. All trigger logic is embedded within the DAG tasks.

## Trigger Logic

The DAG evaluates the following conditions on its weekly schedule:

### 1. Time-Based (Cron)

* **Rule**: Retrain if > 7 days since last `Production` model creation.
* **Config**: `automation.time_interval_days` (default: 7).
* **Purpose**: Ensures model freshness even in stable environments.

### 2. Performance Degradation

* **Rule**: Retrain if current RMSE is > 5% worse than baseline.
* **Config**: `automation.performance_threshold` (default: 5.0%).
* **Source**: MLflow metrics comparison.

### 3. Data Volume

* **Rule**: Retrain if > 10,000 new samples have arrived since last training.
* **Config**: `automation.data_volume_threshold` (default: 10,000).
* **Purpose**: Leverage new information to improve accuracy.

### 4. Data Drift

* **Rule**: Retrain if statistical drift is detected (Evidently (Data and model monitoring library for drift detection using statistical tests.) KS-test p-value \< `drift_threshold`).
* **Config**: `monitoring.drift_threshold` (default: 0.1).
* **Priority**: Marked as **CRITICAL** priority.

## Retraining Workflow

```mermaid theme={null}
graph TD
    %% Styling
    classDef decision fill:#FEF3C7,stroke:#b45309,color:#78350f,stroke-width:2px;
    classDef action fill:#E0E7FF,stroke:#4338ca,color:#1e1b4b,stroke-width:2px;
    classDef data fill:#DCFCE7,stroke:#15803d,color:#14532d,stroke-width:2px;
    classDef alert fill:#FEE2E2,stroke:#b91c1c,color:#7f1d1d,stroke-width:2px;

    Load1[load_reference_data]:::data --> Drift[perform_drift_detection]:::action
    Load2[load_production_data]:::data --> Drift
    Drift --> Check[check_retraining_triggers]:::decision
    Check --> Branch{decide_retraining_branch}:::decision
    Branch -- Retrain --> Train[trigger_retraining]:::action
    Branch -- Skip --> Skip[skip_retraining]:::action
    Train --> Report[generate_drift_report]:::action
    Skip --> Report
    Report --> Alerts[send_alerts]:::alert

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

## DAG Configuration

```python automated_retraining_dag.py theme={null}
@dag(
    schedule="@weekly",
    default_args={
        "owner": "mlops-team",
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
    },
    catchup=False,
)
def automated_retraining_pipeline():
    ...
```

## Priority Handling

The DAG assigns a priority level to the retraining job:

* **CRITICAL**: Immediate retraining (Drift detected). Alert sent via email.
* **HIGH**: Performance issue. Monitor closely.
* **MEDIUM**: New data available. Good to have.
* **LOW**: Routine maintenance.

> [!WARNING]
>
> **Email Alerts**: Critical drift events and retraining triggers send email notifications to the MLOps team, configured via `AIRFLOW__SMTP__*` environment variables in `docker-compose.yml`.

## Post-Detection Tasks

After drift detection, the DAG generates:

1. **Drift Report**: Saved to MinIO (`drift-reports` bucket) as a timestamped JSON, containing per-column drift statistics, trigger decisions, and severity levels.
2. **Email Alerts**: Sent for critical drift events or when retraining is triggered, with detailed summaries of drifted columns and recommended actions.

***

## Related Pages

> [card] [Data Quality & Drift](https://freelance-4888ed08.mintlify.site/ml-solution/data-quality-and-drift.md)
>
> Detailed drift detection methodology and statistical tests.

> [card] [Monitoring DAG](https://freelance-4888ed08.mintlify.site/ml-solution/monitoring-dag.md)
>
> Dataset monitoring pipeline for continuous data quality tracking.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.