Orchestration

Airflow DAGs and Pipeline Automation

Operations

What this does: Coordinates the entire ML workflow with automated scheduling — from data ingestion to model deployment — ensuring every step runs reliably and on time without manual oversight.

The system relies on Apache Airflow 2.10 to orchestrate the end-to-end machine learning workflow.

​

Production Pipeline DAG

The active pipeline (airflow_dag_enhanced.py) integrates robust tracking and persistence mechanisms.

​

Advanced Capabilities

Show Optuna Persistence

Studies are saved to PostgreSQL, allowing optimization to resume after restarts. Multiple workers can contribute to the same study simultaneously.

Show MLflow Tracking

All metrics, parameters, and artifacts are logged to the MLflow Tracking Server for exhaustive experiment auditability.

Show Graceful Degradation

The pipeline includes fallbacks (e.g., using simpler models) if the optimization engine or external databases are unreachable.

​

DAG Configuration

property	value	description
Schedule	Dataset	Triggered when raw_bike_data is updated
Retries	1	Automatic retry with 1-minute delay
Email Failure	Disabled	Alerts currently disabled in code
Owner	kedro	Default pipeline owner

---

​

Kedro Pipelines

While Airflow manages the schedule, Kedro defines the logic. The system is composed of three modular pipelines that transform raw data into production-ready models.

​

Data Processing

• Goal: Sanitize raw inputs and validate schema.
• Key Node: preprocess_data cleans nulls and standardizes the timestamp format.
• Validation: profile_dataset_gx automatically generates a Great Expectations report to catch data quality issues early.

​

Feature Engineering

• Goal: Create predictive signals for the model.
• Key Node: create_derived_features extracts critical signals like hour_of_day, is_weekend, and temperature lags that drive forecast accuracy.

​

Demand Modeling

• Goal: Train, tune, and select the best model.
• Strategy: Challenger/Champion.
  • Challenger: A baseline Random Forest (train_single_model).
  • Champion: A CatBoost Regressor (train_catboost_model) optimized via Optuna.
• Selection: The compare_models node evaluates MAE/RMSE and promotes the winner to the MLflow Registry.

---

​

Dataset Monitoring

Runs independently (@hourly) to ensure data freshness and quality.

1. Check Freshness

   Scans S3 prefixes to verify data arrival within SLAs (e.g., Raw Data < 24h old).

2. Calculate Drift

   Computes the PSI against the reference dataset to detect silent model degradation.

3. Alerting Logic

   • Critical Alerts: Immediate email if critical datasets (Raw Data, Trained Model) are stale.
   • Daily Report: Sent at 08:00 UTC summarizing global system health.
4. Metric Export

   Health metrics are appended to dataset_events.csv for long-term trend analysis.

---

​

Production Hardening

The pipeline is engineered for resilience, not just functionality.

​

Service Level Objectives

Key performance targets hardcoded into the pipeline validation logic:

Metric	Target	Failure Action
Data Completeness	> 50% populated	Pipeline fails fast (ValueError)
RMSE	~51-52	Alert trigger (Performance degradation)
MAE	~30-31	Alert trigger
Freshness	24 Hours	Critical Email Alert

​

Failure Handling

• Automatic Retries: Tasks retry once (retries=1) after a 1-minute backoff to handle transient network blips.
• Optuna Persistence: If the optimization worker crashes, the PostgreSQL backend preserves the study state, allowing the next run to resume without losing progress.
• Graceful Degradation: If MLflow or Optuna DB is unreachable, the pipeline logs a warning but proceeds with local execution to ensure business continuity.

​

Pipeline Configuration

Infrastructure endpoints managed via Airflow Variables and Environment Variables.

Resource	Value/Path	Role
S3 Bucket	s3://demand-forecasting/	Central artifact store
MLflow URI	http://mlflow:5000	Experiment tracking server
Optuna DB	postgresql://.../optuna	Hyperparameter search state
Email	AIRFLOW_ALERT_EMAIL	Critical failure notifications