Testing Strategy

Comprehensive testing framework ensuring production reliability

Engineering

What this does: Validates every component with 50+ automated tests and >85% code coverage — from unit logic to infrastructure health — ensuring production reliability before any deployment.

The system maintains high reliability through a tiered testing strategy covering unit logic, functional correctness, and operational infrastructure. We leverage a Centralized Test Runner to orchestrate over 50+ test cases, ensuring >85% code coverage.

​

Test Suite Overview

Unit & Integration

Validates core logic using pytest. Covers drift detection, optimization, and API endpoints.

Functional Verification

Specialized suite (test_new_features.py) ensuring mathematical correctness of new logic like Feature Lags and CatBoost.

Infrastructure Ops

Verifies operational components: SMTP alerts (test_smtp_alerts.py), DAG integrity, and DB connections.

Data Profiling

Hybrid Strategy: Lightweight Pandas checks for Airflow runtime vs. Deep Great Expectations profiling for development.

​

Running Tests

We use a custom test runner run_tests.py that wraps pytest with advanced reporting and coverage features.

1. Navigate to ML Service

   All testing commands must be run from the app-ml directory:

   cd app-ml
   
2. Execute Test Runner

   Run the comprehensive suite with the custom runner:

   Standard Run

   # Runs standard unit and integration tests
   python run_tests.py
   
3. View Reports

   The runner generates detailed artifacts:

   • HTML Report: test_reports/test_report.html (Summary & Recommendations)
   • Coverage: coverage_report/index.html (Line-by-line analysis)

​

Key Test Categories

​

Production Operations

Critical validation for the automated MLOps layer.

Show Drift Detection Logic

Validates statistical tests used by the DriftDetector:

• Continuous Features: Kolmogorov-Smirnov (KS) Test & Wasserstein Distance.
• Categorical Features: Chi-Square Test & Population Stability Index (PSI).
• Thresholds: Alerts only when P-values drop below 0.05 or drift score > threshold.

Show Retraining Triggers

Ensures the RetrainingScheduler correctly interprets signals:

• Data Growth: Trigger if dataset size increases by >20%.
• Model Decay: Trigger if RMSE exceeds performance threshold (55.0).
• Time Check: Force retraining if model is >7 days old.

​

Functional Verification

Ensures the mathematical and logical correctness of specific features.

• Feature Lags: Verifies correct shift logic (e.g., cnt_lag_1, hr_lag_2) and handling of resulting NaN values.
• CatBoost Integration: Tests model training nodes and the compare_models logic.
• Metrics Parity: Validates the custom compute_zero_safe_mape implementation against manual calculations to prevent division-by-zero errors.

​

Infrastructure Validation

Scripts located in app-ml/scripts/ to verify the environment.

• test_smtp_alerts.py: Sends a test email to verify SMTP credentials and template rendering.
• test_enhanced_dag.py: Parses the Airflow DAG to ensure no import errors or cyclic dependencies exist before deployment.
• verify_ml_integration.py: A “Smoke Test” that checks live connections to PostgreSQL, MinIO, and MLflow.

​

Data Quality Strategy

Runtime Validation (Airflow)

Tool: Pandas & Custom Logic Why: Speed & Low Overhead. Checks: Missing value % < 50%, Empty DataFrame checks, Column existence. File: dags/airflow_dag_enhanced.py

Deep Profiling (Dev)

Tool: Great Expectations Why: Exploratory Data Analysis. Checks: Full distribution analysis, quantile checks, detailed HTML reports. File: test_gx_profiling.py

​

CI/CD Integration

The run_tests.py script returns a non-zero exit code if:

1. Any test fails.
2. Code coverage drops below 85%.

This exit code automatically blocks deployment pipelines in GitHub Actions.