> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Quality & Drift

> Automated data validation, custom profiling, and statistical drift detection.

Data ScienceOperations

> [!NOTE]
>
> **What this does:** Acts as the system's quality guard — automatically detecting when incoming data changes in unexpected ways and flagging potential issues before they degrade prediction accuracy.

## Data Quality Architecture

The system implements a multi-layered approach to data quality, moving from basic schema validation to advanced statistical drift detection.

```mermaid theme={null}
graph TD
    %% Styling
    classDef ingestion fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef validation fill:#E0E7FF,stroke:#4338CA,color:#1E1B4B,stroke-width:2px;
    classDef profiling fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;
    classDef drift fill:#FEF3C7,stroke:#B45309,color:#78350F,stroke-width:2px;
    classDef alert fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:2px;
    classDef success fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;

    Raw["Raw Data<br/><i>Parquet Ingestion</i>"]:::ingestion
    Schema["Pandera Schema<br/><i>Type & Range Checks</i>"]:::validation
    Profile["Custom Profiler<br/><i>Distribution & Stats</i>"]:::profiling
    Drift["Drift Detector<br/><i>KS / PSI / Wasserstein</i>"]:::drift

    Raw --> Schema
    Schema -->|Pass| Profile
    Schema -->|Fail| Reject["Reject & Alert"]:::alert
    Profile --> Drift
    Drift -->|No Drift| Healthy["Healthy Pipeline"]:::success
    Drift -->|Drift Detected| Retrain["Trigger Retraining"]:::alert

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

## Schema Validation (Pandera)

We use [Pandera](https://pandera.readthedocs.io/) to enforce strict type checking and statistical bounds on the raw data immediately upon loading.

**Schema Definition**: `src/demand_forecasting/pipelines/data_processing/schemas.py`

```python theme={null}
class RawBikeDataSchema(pa.DataFrameModel):
    # Numeric ranges
    season: Series[int] = pa.Field(ge=1, le=4)
    yr: Series[int] = pa.Field(ge=0, le=1)
    
    # Normalized floats
    temp: Series[float] = pa.Field(ge=0.0, le=1.0)
    
    # Target integrity
    cnt: Series[int] = pa.Field(ge=0)
```

## Custom Data Profiler

Instead of relying solely on heavy external libraries, we use a lightweight, custom profiler (`src/demand_forecasting/utils/gx_profiler_simple.py`) inspired by Great Expectations but optimized for speed and HTML reporting.

### Generated Reports

The profiler automatically generates an HTML report (`dataset_profile_YYYYMMDD.html`) for every pipeline run, containing:

* **Missing Value Analysis**: Heatmaps and percentage counts.
* **Distribution Statistics**: Mean, std, min, max, and quantiles for all numeric features.
* **Categorical Summaries**: Frequency counts and cardinality checks.
* **Quality Flags**: Automatic warnings for "High Cardinality" or "High Warning Rate".

These reports are saved to `data/docs/` and can be served via a static web server or viewed directly.

## Drift Detection System

Drift detection is handled by `scripts/production_ops/detect_data_drift.py`. It persists detection history to a PostgreSQL table `drift_records`, allowing for trend analysis over time.

### Statistical Tests Used

The system employs four distinct statistical tests to ensure robust drift detection across different data types:

| Test | Target Data Type | Purpose | Threshold |
| :- | :- | :- | :- |
| **KS-Test** (Kolmogorov-Smirnov) | Continuous | Compares cumulative distribution functions. Sensitive to shape changes. | p-value \< 0.05 |
| **PSI** (Population Stability Index) | Continuous & Categorical | Measures the magnitude of shift. Industry standard for financial/risk models. | PSI > 0.2 |
| **Wasserstein Distance** | Continuous | Measures the "work" needed to transform one distribution into another. Robust to noise. | Dist > 0.1 |
| **Chi-Square Test** | Categorical | Tests independence of frequencies between reference and production data. | p-value \< 0.05 |

### Drift Alerting

If **ANY** of these tests fail for a feature, a "Drift Detected" event is logged. If critical features (like `temp` or `hr`) drift, an email alert is immediately sent to the MLOps team via the configured SMTP server.

## Database Schema: `drift_records`

Drift history is stored in PostgreSQL for long-term monitoring:

```sql theme={null}
CREATE TABLE drift_records (
    id SERIAL PRIMARY KEY,
    check_timestamp TIMESTAMP,
    feature_name VARCHAR(255),
    drift_detected BOOLEAN,
    drift_score FLOAT,           -- Max score (PSI or 1-p_value)
    test_method VARCHAR(50),     -- e.g., 'ks_test', 'psi'
    production_mean FLOAT,
    reference_mean FLOAT
);
```

You can query this table to visualize "Drift Over Time" dashboards in Grafana or Superset.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.