Monitoring & Alerting

Dedicated production monitoring for dataset freshness and system health.

Operations

What this does: Runs a dedicated health-check pipeline that monitors data freshness independently of model training — so data outages are caught immediately, even if the training pipeline is paused.

​

Overview

While the training pipeline (airflow_dag_enhanced) handles model updates, a separate, dedicated monitoring pipeline (dataset_monitoring_dag.py) runs continuously to ensure data health.

Separating monitoring from training ensures that even if the training pipeline fails, validation checks continue to run and potential data outages are detected immediately.

​

Monitoring DAG Architecture

The dataset_monitoring_dag runs on an hourly schedule (Schedule: 0 * * * *) and performs the following checks:

​

1. S3 Timestamp Validation

The system does not just check “is data present?” but “how stale is it?”.

• It queries the S3 bucket (demand-forecasting) directly.
• Compares the LastModified timestamp of key files against a configured max_age_hours threshold (e.g., 24h for raw data, 48h for models).

​

2. Metric Reporting

It generates a daily HTML email report sent at 8 AM UTC, summarizing:

• Freshness: Which datasets were updated in the last 24h.
• Failures: Any validation checks that failed.
• Trend: Sparklines or logic indicating if data arrival is delayed compared to historical norms.

​

Critical vs. Non-Critical Alerts

The system distinguishes between dataset importance to avoid “alert fatigue”:

Dataset Type	Alert Policy	Example
Critical	Immediate Email + Slack (via webhook)	01_raw/bikes.csv (Source data)
Non-Critical	Log Warning Only	08_reporting/summary.json

​

Code Reference

The freshness logic is defined in dataset_monitoring_dag.py:

def check_dataset_freshness(dataset_uri: str, max_age_hours: int = 24) -> Tuple[bool, Optional[float]]:
    # ... checks S3 timestamp ...
    if hours_since_update > max_age_hours:
        return False, hours_since_update
    return True, hours_since_update

​

Email Reporting Template

The system uses a Jinja2 template for its HTML reports, ensuring they are readable on mobile devices.

• Green Status: All critical datasets fresh.
• Red Status: At least one critical dataset stale > 24h.