> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring & Alerting

> Dedicated production monitoring for dataset freshness and system health.

Operations

> [!NOTE]
>
> **What this does:** Runs a dedicated health-check pipeline that monitors data freshness independently of model training — so data outages are caught immediately, even if the training pipeline is paused.

## Overview

While the training pipeline (`airflow_dag_enhanced`) handles model updates, a separate, dedicated monitoring pipeline (`dataset_monitoring_dag.py`) runs continuously to ensure data health.

Separating monitoring from training ensures that **even if the training pipeline fails**, validation checks continue to run and potential data outages are detected immediately.

## Monitoring DAG Architecture

```mermaid theme={null}
graph TD
    %% Styling
    classDef trigger fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;
    classDef check fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef decision fill:#FEF3C7,stroke:#B45309,color:#78350F,stroke-width:2px;
    classDef alert fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:2px;
    classDef ok fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;
    classDef report fill:#E0E7FF,stroke:#4338CA,color:#1E1B4B,stroke-width:2px;

    Cron["Hourly Trigger<br/><i>0 * * * *</i>"]:::trigger
    S3["S3 Freshness Check<br/><i>LastModified Scan</i>"]:::check
    Classify{Critical<br/>Dataset?}:::decision

    Cron --> S3
    S3 --> Classify

    Classify -->|Yes| Stale{"Stale > 24h?"}:::decision
    Classify -->|No| LogOnly["Log Warning<br/><i>Non-Critical</i>"]:::ok

    Stale -->|Yes| EmailAlert["Immediate Email<br/><i>Critical Alert</i>"]:::alert
    Stale -->|No| Fresh["Fresh & Healthy"]:::ok

    EmailAlert --> DailyReport["Daily Summary<br/><i>08:00 UTC</i>"]:::report
    Fresh --> DailyReport
    LogOnly --> DailyReport

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

The `dataset_monitoring_dag` runs on an **hourly schedule** (Schedule: `0 * * * *`) and performs the following checks:

### 1. S3 Timestamp Validation

The system does not just check "is data present?" but "how stale is it?".

* It queries the S3 bucket (`demand-forecasting`) directly.
* Compares the `LastModified` timestamp of key files against a configured `max_age_hours` threshold (e.g., **24h** for raw data, **48h** for models).

### 2. Metric Reporting

It generates a daily HTML email report sent at **8 AM UTC**, summarizing:

* **Freshness**: Which datasets were updated in the last 24h.
* **Failures**: Any validation checks that failed.
* **Trend**: Sparklines or logic indicating if data arrival is delayed compared to historical norms.

## Critical vs. Non-Critical Alerts

The system distinguishes between dataset importance to avoid "alert fatigue":

| Dataset Type | Alert Policy | Example |
| :- | :- | :- |
| **Critical** | Immediate Email + Slack (via webhook) | `01_raw/bikes.csv` (Source data) |
| **Non-Critical** | Log Warning Only | `08_reporting/summary.json` |

## Code Reference

The freshness logic is defined in `dataset_monitoring_dag.py`:

```python theme={null}
def check_dataset_freshness(dataset_uri: str, max_age_hours: int = 24) -> Tuple[bool, Optional[float]]:
    # ... checks S3 timestamp ...
    if hours_since_update > max_age_hours:
        return False, hours_since_update
    return True, hours_since_update
```

## Email Reporting Template

The system uses a Jinja2 template for its HTML reports, ensuring they are readable on mobile devices.

* **Green Status**: All critical datasets fresh.
* **Red Status**: At least one critical dataset stale > 24h.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.