Data Pipeline

How raw bike-sharing data is validated with Pandera schemas, cleaned, and prepared for feature engineering

EngineeringData Science

What this does: Automatically cleans and validates incoming bike-sharing data so that downstream forecasts are always based on accurate, trustworthy inputs — reducing data errors before they reach the model.

The Data Pipeline is responsible for ingesting raw data, validating its schema with Pandera, and cleaning it for downstream consumption.

​

Pipeline Flow

The pipeline uses a defensive programming approach — validation issues generate warnings but processing continues with available data, rather than failing fast.

​

Node: validate_etl_data

The primary validation function checks data quality at each ETL stage.

1. Dataset Size Checks

   Validates that the dataset is non-empty, has at least 100 samples for reliable training, and contains a minimum of 5 feature columns.

2. Required Columns

   Ensures all 15 required columns are present: season, yr, mnth, hr, holiday, weekday, workingday, weathersit, temp, atemp, hum, windspeed, casual, registered, cnt.

3. Range & Outlier Detection

   • Temperature & Humidity: Validates normalized [0, 1] range
   • IQR Outliers: Flags statistical outliers using the Interquartile Range method
   • Skewness: Warns if any numeric column has skewness > 2.0

​

Node: preprocess_data

This node ensures the dataset is clean and stateless before feature engineering.

1. Column Removal

   Drops administrative columns that offer no predictive value:

   • yr (Year — does not help with predictions)
   • casual & registered (Leakage — component parts of target cnt)
   • datetime (Not needed for model features)
2. Schema Validation

   Validates input against strict Pandera schemas (RawBikeDataSchema) to prevent type errors.

3. Defensive Dropping

   Only drops columns that actually exist — logs warnings for any expected columns that are missing, indicating upstream pipeline changes.

​

Code Snippet

nodes.py

@pa.check_types
def preprocess_data(df: DataFrame[RawBikeDataSchema]) -> pd.DataFrame:
    """Preprocess data by dropping columns not needed for modeling."""
    # Validate data quality with defensive approach
    if not validate_etl_data(df, "preprocessing"):
        logger.warning(
            "Data validation issues detected. "
            "Proceeding with available data."
        )

    drop_cols = ["yr", "casual", "registered", "datetime"]
    existing_cols = [col for col in drop_cols if col in df.columns]
    result = df.drop(columns=existing_cols)

    logger.info(
        f"Preprocessing complete. Dropped {len(existing_cols)} columns. "
        f"Output shape: {result.shape}."
    )
    return result

​

Data Catalog

Dataset	Type	Path
raw_bike_data	pandas.ParquetDataset	data/01_raw/raw_bike_data.parquet
preprocessed_bike_data	pandas.ParquetDataset	data/02_intermediate/preprocessed_bike_data.parquet
temporal_features	pandas.ParquetDataset	data/02_intermediate/temporal_features.parquet
engineered_features	pandas.ParquetDataset	data/02_intermediate/engineered_features.parquet
model_input_table	pandas.ParquetDataSet	data/03_primary/model_input_table.parquet

---

​

Related Pages

Feature Engineering

How preprocessed data is transformed into lag features and split for training.

Data Catalog

Complete registry of all datasets, schemas, and storage paths.