> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Testing Strategy

> Comprehensive testing framework ensuring production reliability

Engineering

> [!NOTE]
>
> **What this does:** Validates every component with 50+ automated tests and >85% code coverage — from unit logic to infrastructure health — ensuring production reliability before any deployment.

The system maintains high reliability through a tiered testing strategy covering unit logic, functional correctness, and operational infrastructure. We leverage a **Centralized Test Runner** to orchestrate over 50+ test cases, ensuring **>85% code coverage**.

```mermaid theme={null}
graph BT
    %% Styling
    classDef unit fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;
    classDef integration fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef infra fill:#FEF3C7,stroke:#B45309,color:#78350F,stroke-width:2px;
    classDef profiling fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;

    Unit["Unit Tests<br/><i>Drift, Optimization, API Logic</i>"]:::unit
    Int["Integration Tests<br/><i>Feature Lags, CatBoost, Metrics</i>"]:::integration
    Infra["Infrastructure Tests<br/><i>SMTP, DAG Parsing, DB Connections</i>"]:::infra
    Profile["Data Profiling<br/><i>Pandas Runtime + GX Deep Analysis</i>"]:::profiling

    Unit --> Int --> Infra --> Profile

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

## Test Suite Overview

> [card] Unit & Integration
>
> Validates core logic using `pytest`. Covers drift detection, optimization, and API endpoints.

> [card] Functional Verification
>
> Specialized suite (`test_new_features.py`) ensuring mathematical correctness of new logic like Feature Lags and CatBoost.

> [card] Infrastructure Ops
>
> Verifies operational components: SMTP alerts (`test_smtp_alerts.py`), DAG integrity, and DB connections.

> [card] Data Profiling
>
> **Hybrid Strategy**: Lightweight Pandas checks for Airflow runtime vs. Deep Great Expectations profiling for development.

## Running Tests

We use a custom test runner `run_tests.py` that wraps `pytest` with advanced reporting and coverage features.

**1. Navigate to ML Service**

All testing commands must be run from the `app-ml` directory:

```bash theme={null}
cd app-ml
```

**2. Execute Test Runner**

Run the comprehensive suite with the custom runner:

```bash Standard Run theme={null}
# Runs standard unit and integration tests
python run_tests.py
```

```bash Full Suite theme={null}
# Includes slow tests (e.g., optimization) and external integrations
python run_tests.py --include-slow --include-mlflow --include-optuna
```

**3. View Reports**

The runner generates detailed artifacts:

* **HTML Report**: `test_reports/test_report.html` (Summary & Recommendations)
* **Coverage**: `coverage_report/index.html` (Line-by-line analysis)

## Key Test Categories

### Production Operations

Critical validation for the automated MLOps layer.

**Drift Detection Logic**

Validates statistical tests used by the `DriftDetector`:

* **Continuous Features**: Kolmogorov-Smirnov (KS) Test & Wasserstein Distance.
* **Categorical Features**: Chi-Square Test & Population Stability Index (PSI).
* **Thresholds**: Alerts only when P-values drop below `0.05` or drift score > threshold.

**Retraining Triggers**

Ensures the `RetrainingScheduler` correctly interprets signals:

* **Data Growth**: Trigger if dataset size increases by >20%.
* **Model Decay**: Trigger if RMSE exceeds performance threshold (55.0).
* **Time Check**: Force retraining if model is >7 days old.

### Functional Verification

Ensures the mathematical and logical correctness of specific features.

* **Feature Lags**: Verifies correct shift logic (e.g., `cnt_lag_1`, `hr_lag_2`) and handling of resulting `NaN` values.
* **CatBoost Integration**: Tests model training nodes and the `compare_models` logic.
* **Metrics Parity**: Validates the custom `compute_zero_safe_mape` implementation against manual calculations to prevent division-by-zero errors.

### Infrastructure Validation

Scripts located in `app-ml/scripts/` to verify the environment.

* **`test_smtp_alerts.py`**: Sends a test email to verify SMTP credentials and template rendering.
* **`test_enhanced_dag.py`**: Parses the Airflow DAG to ensure no import errors or cyclic dependencies exist before deployment.
* **`verify_ml_integration.py`**: A "Smoke Test" that checks live connections to PostgreSQL, MinIO, and MLflow.

## Data Quality Strategy

> [card] Runtime Validation (Airflow)
>
> **Tool**: Pandas & Custom Logic
> **Why**: Speed & Low Overhead.
> **Checks**: Missing value % \< 50%, Empty DataFrame checks, Column existence.
> **File**: `dags/airflow_dag_enhanced.py`

> [card] Deep Profiling (Dev)
>
> **Tool**: Great Expectations
> **Why**: Exploratory Data Analysis.
> **Checks**: Full distribution analysis, quantile checks, detailed HTML reports.
> **File**: `test_gx_profiling.py`

## CI/CD Integration

The `run_tests.py` script returns a non-zero exit code if:

1. Any test fails.
2. Code coverage drops below **85%**.

This exit code automatically blocks deployment pipelines in GitHub Actions.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.