> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations & Reliability

> System health, automated recovery, and enterprise-grade alerting

Operations

> [!NOTE]
>
> **What this does:** Monitors system health 24/7 and automatically alerts on anomalies — providing enterprise-grade uptime and rapid incident response to keep forecasts flowing without interruption.

The system is engineered for continuous reliability, providing automated monitoring, lifecycle management, and resilient failure recovery for mission-critical demand forecasts.

## Model Lifecycle Management

The system automatically manages the transition of models through the MLOps lifecycle, ensuring only validated models reach the production environment.

```mermaid theme={null}
graph LR
    %% Styling
    classDef training fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef candidate fill:#E0E7FF,stroke:#4338CA,color:#1E1B4B,stroke-width:2px;
    classDef validation fill:#FEF3C7,stroke:#B45309,color:#78350F,stroke-width:2px;
    classDef promotion fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;
    classDef production fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;
    classDef failure fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:2px;

    Train["Training<br/><i>CatBoost + Optuna</i>"]:::training
    Candidate["Candidate<br/><i>regressor_YYYYMMDD</i>"]:::candidate
    Validate{"Validate<br/><i>vs Baseline</i>"}:::validation
    Promote["Promote<br/><i>MLflow + MinIO</i>"]:::promotion
    Serve["Production<br/><i>FastAPI Serving</i>"]:::production
    Reject["Reject<br/><i>Keep Current</i>"]:::failure

    Train --> Candidate --> Validate
    Validate -->|Better| Promote --> Serve
    Validate -->|Worse| Reject

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

### Model Promotion

**1. Candidate Selection**

Identifies the latest training run artifact (`regressor_YYYYMMDD.pickle`) from the registry.

**2. Validation**

Compares performance metrics against the current production baseline.

**3. Promotion**

Updates the production alias in MinIO and MLflow for seamless serving switchover.

### Artifact Registry

All models and metadata are persistently stored in **MinIO (S3)** and **PostgreSQL**, providing a full audit trail of every prediction ever served.

## System Resilience

The platform is designed for **graceful degradation**, ensuring forecasts are delivered even if secondary services are offline.

**Isolated Failure Domains**

Failures in experiment tracking (MLflow) or hyperparameter storage (Optuna) do not block core training or inference.

**Graceful Fallback**

Use pre-validated stable configurations if the optimization engine is unreachable.

**Idempotency**

All operations are idempotent, allowing for safe retries and self-healing.

## Production Alerting

To support rapid incident response, the platform utilizes **structured HTML email notifications** for immediate triage and visibility.

* **Automated Alerts**: Real-time notifications for pipeline failures, data drift, or service outages.
* **Context-Rich Content**: Includes failure severity, model versions, run IDs, and direct links to Airflow/MLflow logs.
* **Triage Safety**: Designed to separate actionable signals from noisy warnings, ensuring the MLOps team can respond to critical events immediately.

## Monitoring & Drift Detection

We compare live data against the training baseline to detect Data Drift (The change in the distribution of input data, which can lead to model performance degradation.).

### Drift Metrics

**PSI (Population Stability Index)**

**Threshold**: `> 0.2`

Measures major shifts in overall population distribution.

**Wasserstein Distance**

**Threshold**: `> 0.1`

Sensitive to changes in distribution shape and outliers.

> [!TIP]
>
> **Automated Recovery**: The retraining scheduler initiates self-healing if drift is detected or performance decays below thresholds.

## Infrastructure & Logging

### Artifact Storage

The `create_log_bucket.py` script ensures our MinIO artifact store is initialized.

> [!WARNING]
>
> **Idempotency**: The script checks `BucketAlreadyOwnedByYou` to prevent errors on restart or redeployment.

### Centralized Logging

Centralized logging configuration (`logging_config.py`) suppresses noise from secondary libraries (Kedro, Great Expectations), ensuring that the MLOps team can focus on proprietary application signals and critical system events.

### Validation Logic

Drift and retraining logic are validated using standard `pytest` suites:

```python theme={null}
# Analysis from test_production_ops.py
def test_drift_trigger(self):
    detector = DriftDetector(reference_data)
    drift_score = detector.calculate_psi(current_data)
    assert drift_score > 0.2  # Critical threshold
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.