> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Pipeline

> How raw bike-sharing data is validated with Pandera schemas, cleaned, and prepared for feature engineering

EngineeringData Science

> [!NOTE]
>
> **What this does:** Automatically cleans and validates incoming bike-sharing data so that downstream forecasts are always based on accurate, trustworthy inputs — reducing data errors before they reach the model.

The Data Pipeline is responsible for ingesting raw data, validating its schema with Pandera (Statistical data validation library for pandas DataFrames.), and cleaning it for downstream consumption.

## Pipeline Flow

```mermaid theme={null}
graph TD
    %% Styling
    classDef process fill:#E0E7FF,stroke:#4338ca,color:#1e1b4b,stroke-width:2px;
    classDef decision fill:#FEF3C7,stroke:#b45309,color:#78350f,stroke-width:2px;
    classDef storage fill:#DCFCE7,stroke:#15803d,color:#14532d,stroke-width:2px;
    classDef warn fill:#FEF9C3,stroke:#ca8a04,color:#713f12,stroke-width:2px;

    Raw[(Raw Parquet)]:::storage -->|Load| Schema{Pandera Schema<br/><i>RawBikeDataSchema</i>}:::decision
    Schema -->|Pass| ETL[validate_etl_data]:::process
    Schema -->|Fail| Reject[Reject Record]:::warn
    ETL -->|Issues?| Warn[Log Warning & Continue]:::warn
    ETL -->|Clean| Clean[preprocess_data]:::process
    Clean -->|Save| Inter[(Intermediate Parquet)]:::storage

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

> [!NOTE]
>
> The pipeline uses a **defensive programming** approach — validation issues generate warnings but processing continues with available data, rather than failing fast.

## Node: `validate_etl_data`

The primary validation function checks data quality at each ETL stage.

**1. Dataset Size Checks**

Validates that the dataset is non-empty, has at least **100 samples** for reliable training, and contains a minimum of **5 feature columns**.

**2. Required Columns**

Ensures all 15 required columns are present: `season`, `yr`, `mnth`, `hr`, `holiday`, `weekday`, `workingday`, `weathersit`, `temp`, `atemp`, `hum`, `windspeed`, `casual`, `registered`, `cnt`.

**3. Range & Outlier Detection**

* **Temperature & Humidity**: Validates normalized `[0, 1]` range
* **IQR Outliers**: Flags statistical outliers using the Interquartile Range method
* **Skewness**: Warns if any numeric column has skewness > 2.0

## Node: `preprocess_data`

This node ensures the dataset is clean and stateless before feature engineering.

**1. Column Removal**

Drops administrative columns that offer no predictive value:

* `yr` (Year — does not help with predictions)
* `casual` & `registered` (Leakage — component parts of target `cnt`)
* `datetime` (Not needed for model features)

**2. Schema Validation**

Validates input against strict `Pandera` schemas (`RawBikeDataSchema`) to prevent type errors.
\[!NOTE] Validation occurs **before** column dropping using the `@pa.check_types` decorator.

**3. Defensive Dropping**

Only drops columns that actually exist — logs warnings for any expected columns that are missing, indicating upstream pipeline changes.

### Code Snippet

```python nodes.py theme={null}
@pa.check_types
def preprocess_data(df: DataFrame[RawBikeDataSchema]) -> pd.DataFrame:
    """Preprocess data by dropping columns not needed for modeling."""
    # Validate data quality with defensive approach
    if not validate_etl_data(df, "preprocessing"):
        logger.warning(
            "Data validation issues detected. "
            "Proceeding with available data."
        )

    drop_cols = ["yr", "casual", "registered", "datetime"]
    existing_cols = [col for col in drop_cols if col in df.columns]
    result = df.drop(columns=existing_cols)

    logger.info(
        f"Preprocessing complete. Dropped {len(existing_cols)} columns. "
        f"Output shape: {result.shape}."
    )
    return result
```

## Data Catalog

| Dataset | Type | Path |
| :- | :- | :- |
| `raw_bike_data` | `pandas.ParquetDataset` | `data/01_raw/raw_bike_data.parquet` |
| `preprocessed_bike_data` | `pandas.ParquetDataset` | `data/02_intermediate/preprocessed_bike_data.parquet` |
| `temporal_features` | `pandas.ParquetDataset` | `data/02_intermediate/temporal_features.parquet` |
| `engineered_features` | `pandas.ParquetDataset` | `data/02_intermediate/engineered_features.parquet` |
| `model_input_table` | `pandas.ParquetDataSet` | `data/03_primary/model_input_table.parquet` |

***

## Related Pages

> [card] [Feature Engineering](https://freelance-4888ed08.mintlify.site/ml-solution/feature-engineering.md)
>
> How preprocessed data is transformed into lag features and split for training.

> [card] [Data Catalog](https://freelance-4888ed08.mintlify.site/configuration/data-catalog.md)
>
> Complete registry of all datasets, schemas, and storage paths.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.