Data Quality & Drift

Automated data validation, custom profiling, and statistical drift detection.

Data ScienceOperations

What this does: Acts as the system’s quality guard — automatically detecting when incoming data changes in unexpected ways and flagging potential issues before they degrade prediction accuracy.

​

Data Quality Architecture

The system implements a multi-layered approach to data quality, moving from basic schema validation to advanced statistical drift detection.

​

Schema Validation (Pandera)

We use Pandera to enforce strict type checking and statistical bounds on the raw data immediately upon loading.

Schema Definition: src/demand_forecasting/pipelines/data_processing/schemas.py

class RawBikeDataSchema(pa.DataFrameModel):
    # Numeric ranges
    season: Series[int] = pa.Field(ge=1, le=4)
    yr: Series[int] = pa.Field(ge=0, le=1)
    
    # Normalized floats
    temp: Series[float] = pa.Field(ge=0.0, le=1.0)
    
    # Target integrity
    cnt: Series[int] = pa.Field(ge=0)

​

Custom Data Profiler

Instead of relying solely on heavy external libraries, we use a lightweight, custom profiler (src/demand_forecasting/utils/gx_profiler_simple.py) inspired by Great Expectations but optimized for speed and HTML reporting.

​

Generated Reports

The profiler automatically generates an HTML report (dataset_profile_YYYYMMDD.html) for every pipeline run, containing:

• Missing Value Analysis: Heatmaps and percentage counts.
• Distribution Statistics: Mean, std, min, max, and quantiles for all numeric features.
• Categorical Summaries: Frequency counts and cardinality checks.
• Quality Flags: Automatic warnings for “High Cardinality” or “High Warning Rate”.

These reports are saved to data/docs/ and can be served via a static web server or viewed directly.

​

Drift Detection System

Drift detection is handled by scripts/production_ops/detect_data_drift.py. It persists detection history to a PostgreSQL table drift_records, allowing for trend analysis over time.

​

Statistical Tests Used

The system employs four distinct statistical tests to ensure robust drift detection across different data types:

Test	Target Data Type	Purpose	Threshold
KS-Test (Kolmogorov-Smirnov)	Continuous	Compares cumulative distribution functions. Sensitive to shape changes.	p-value < 0.05
PSI (Population Stability Index)	Continuous & Categorical	Measures the magnitude of shift. Industry standard for financial/risk models.	PSI > 0.2
Wasserstein Distance	Continuous	Measures the “work” needed to transform one distribution into another. Robust to noise.	Dist > 0.1
Chi-Square Test	Categorical	Tests independence of frequencies between reference and production data.	p-value < 0.05

​

Drift Alerting

If ANY of these tests fail for a feature, a “Drift Detected” event is logged. If critical features (like temp or hr) drift, an email alert is immediately sent to the MLOps team via the configured SMTP server.

​

Database Schema: drift_records

Drift history is stored in PostgreSQL for long-term monitoring:

CREATE TABLE drift_records (
    id SERIAL PRIMARY KEY,
    check_timestamp TIMESTAMP,
    feature_name VARCHAR(255),
    drift_detected BOOLEAN,
    drift_score FLOAT,           -- Max score (PSI or 1-p_value)
    test_method VARCHAR(50),     -- e.g., 'ks_test', 'psi'
    production_mean FLOAT,
    reference_mean FLOAT
);

You can query this table to visualize “Drift Over Time” dashboards in Grafana or Superset.