Automated Retraining

Automated weekly retraining DAG with 4 triggers: time-based, performance, data volume, and drift detection

OperationsData Science

What this does: The system continuously monitors prediction quality and retrains itself weekly when it detects degradation — ensuring forecasts stay accurate as real-world conditions change, without human intervention.

​

Overview

Models degrade over time. The system employs an automated Airflow DAG (dags/automated_retraining_dag.py) that runs weekly and monitors 4 distinct triggers to decide when to launch a new training job.

The retraining pipeline runs as a @taskflow decorated Airflow DAG, not a standalone Python class. All trigger logic is embedded within the DAG tasks.

​

Trigger Logic

The DAG evaluates the following conditions on its weekly schedule:

​

1. Time-Based (Cron)

• Rule: Retrain if > 7 days since last Production model creation.
• Config: automation.time_interval_days (default: 7).
• Purpose: Ensures model freshness even in stable environments.

​

2. Performance Degradation

• Rule: Retrain if current RMSE is > 5% worse than baseline.
• Config: automation.performance_threshold (default: 5.0%).
• Source: MLflow metrics comparison.

​

3. Data Volume

• Rule: Retrain if > 10,000 new samples have arrived since last training.
• Config: automation.data_volume_threshold (default: 10,000).
• Purpose: Leverage new information to improve accuracy.

​

4. Data Drift

• Rule: Retrain if statistical drift is detected (Evidently KS-test p-value < drift_threshold).
• Config: monitoring.drift_threshold (default: 0.1).
• Priority: Marked as CRITICAL priority.

​

Retraining Workflow

​

DAG Configuration

automated_retraining_dag.py

@dag(
    schedule="@weekly",
    default_args={
        "owner": "mlops-team",
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
    },
    catchup=False,
)
def automated_retraining_pipeline():
    ...

​

Priority Handling

The DAG assigns a priority level to the retraining job:

• CRITICAL: Immediate retraining (Drift detected). Alert sent via email.
• HIGH: Performance issue. Monitor closely.
• MEDIUM: New data available. Good to have.
• LOW: Routine maintenance.

Email Alerts: Critical drift events and retraining triggers send email notifications to the MLOps team, configured via AIRFLOW__SMTP__* environment variables in docker-compose.yml.

​

Post-Detection Tasks

After drift detection, the DAG generates:

1. Drift Report: Saved to MinIO (drift-reports bucket) as a timestamped JSON, containing per-column drift statistics, trigger decisions, and severity levels.
2. Email Alerts: Sent for critical drift events or when retraining is triggered, with detailed summaries of drifted columns and recommended actions.

---

​

Related Pages

Data Quality & Drift

Detailed drift detection methodology and statistical tests.

Monitoring DAG

Dataset monitoring pipeline for continuous data quality tracking.