> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Orchestration

> Airflow DAGs and Pipeline Automation

Operations

> [!NOTE]
>
> **What this does:** Coordinates the entire ML workflow with automated scheduling — from data ingestion to model deployment — ensuring every step runs reliably and on time without manual oversight.

The system relies on **Apache Airflow 2.10** to orchestrate the end-to-end machine learning workflow.

## Production Pipeline DAG

The active pipeline (`airflow_dag_enhanced.py`) integrates robust tracking and persistence mechanisms.

```mermaid theme={null}
graph TD
    %% Styling
    classDef validate fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef preprocess fill:#E0E7FF,stroke:#4338CA,color:#1E1B4B,stroke-width:2px;
    classDef feature fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;
    classDef train fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;

    A["Validate Raw Data<br/><i>Schema & Integrity</i>"]:::validate --> B["Preprocess Data<br/><i>Clean & Transform</i>"]:::preprocess
    B --> C["Feature Engineering<br/><i>Lags & Signals</i>"]:::feature
    C --> D["Train Optimized Model<br/><i>CatBoost + Optuna</i>"]:::train
    
    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

### Advanced Capabilities

**Optuna Persistence**

Studies are saved to PostgreSQL, allowing optimization to resume after restarts. Multiple workers can contribute to the same study simultaneously.

**MLflow Tracking**

All metrics, parameters, and artifacts are logged to the MLflow Tracking Server for exhaustive experiment auditability.

**Graceful Degradation**

The pipeline includes fallbacks (e.g., using simpler models) if the optimization engine or external databases are unreachable.

### DAG Configuration

| property | value | description |
| :- | :- | :- |
| **Schedule** | `Dataset` | Triggered when `raw_bike_data` is updated |
| **Retries** | `1` | Automatic retry with 1-minute delay |
| **Email Failure** | `Disabled` | Alerts currently disabled in code |
| **Owner** | `kedro` | Default pipeline owner |

***

## Kedro Pipelines

While Airflow manages the schedule, **Kedro** defines the logic. The system is composed of three modular pipelines that transform raw data into production-ready models.

```mermaid theme={null}
graph TD
    %% Styling
    classDef pipeline fill:#F3F4F6,stroke:#4B5563,color:#1F2937,stroke-width:2px;
    classDef node fill:#EEF2FF,stroke:#6366F1,color:#1E1B4B,stroke-width:1px;

    subgraph DP ["Data Processing"]
        P1["Preprocess Data"]:::node
        P2["Profile DB (GX)"]:::node
        P1 --> P2
    end

    subgraph FE ["Feature Engineering"]
        F1["Time Features"]:::node
        F2["Lag Features"]:::node
        F1 --> F2
    end

    subgraph DM ["Demand Modeling"]
        M1["Train Candidate (RF)"]:::node
        M2["Optimize Champion (CatBoost)"]:::node
        M3["Compare & Select"]:::node
        M1 --> M3
        M2 --> M3
    end

    DP --> FE --> DM

    class DP,FE,DM pipeline
```

### Data Processing

* **Goal**: Sanitize raw inputs and validate schema.
* **Key Node**: `preprocess_data` cleans nulls and standardizes the timestamp format.
* **Validation**: `profile_dataset_gx` automatically generates a Great Expectations report to catch data quality issues early.

### Feature Engineering

* **Goal**: Create predictive signals for the model.
* **Key Node**: `create_derived_features` extracts critical signals like `hour_of_day`, `is_weekend`, and temperature lags that drive forecast accuracy.

### Demand Modeling

* **Goal**: Train, tune, and select the best model.
* **Strategy**: Challenger/Champion.
  * **Challenger**: A baseline Random Forest (`train_single_model`).
  * **Champion**: A CatBoost Regressor (`train_catboost_model`) optimized via Optuna.
* **Selection**: The `compare_models` node evaluates MAE/RMSE and promotes the winner to the MLflow Registry.

***

## Dataset Monitoring

Runs independently (`@hourly`) to ensure data freshness and quality.

**1. Check Freshness**

Scans S3 prefixes to verify data arrival within SLAs (e.g., **Raw Data \< 24h old**).

**2. Calculate Drift**

Computes the PSI (Population Stability Index: A metric to measure how much the distribution of a variable has changed.) against the reference dataset to detect silent model degradation.

**3. Alerting Logic**

* **Critical Alerts**: Immediate email if critical datasets (Raw Data, Trained Model) are stale.
* **Daily Report**: Sent at **08:00 UTC** summarizing global system health.

**4. Metric Export**

Health metrics are appended to `dataset_events.csv` for long-term trend analysis.

***

## Production Hardening

The pipeline is engineered for resilience, not just functionality.

### Service Level Objectives

Key performance targets hardcoded into the pipeline validation logic:

| Metric | Target | Failure Action |
| :- | :- | :- |
| **Data Completeness** | > 50% populated | Pipeline fails fast (ValueError) |
| **RMSE** | \~51-52 | Alert trigger (Performance degradation) |
| **MAE** | \~30-31 | Alert trigger |
| **Freshness** | 24 Hours | Critical Email Alert |

### Failure Handling

* **Automatic Retries**: Tasks retry once (`retries=1`) after a 1-minute backoff to handle transient network blips.
* **Optuna Persistence**: If the optimization worker crashes, the PostgreSQL backend preserves the study state, allowing the next run to resume without losing progress.
* **Graceful Degradation**: If MLflow or Optuna DB is unreachable, the pipeline logs a warning but proceeds with local execution to ensure business continuity.

### Pipeline Configuration

Infrastructure endpoints managed via Airflow Variables and Environment Variables.

| Resource | Value/Path | Role |
| :- | :- | :- |
| **S3 Bucket** | `s3://demand-forecasting/` | Central artifact store |
| **MLflow URI** | `http://mlflow:5000` | Experiment tracking server |
| **Optuna DB** | `postgresql://.../optuna` | Hyperparameter search state |
| **Email** | `AIRFLOW_ALERT_EMAIL` | Critical failure notifications |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.