Data Catalog

Dataset Definitions and Storage Formats

The Data Catalog (conf/base/catalog.yml) serves as the single source of truth for all data assets in the system.

​

Data Layers

We follow the Kedro data engineering convention:

Layer	Path Pattern	Format	Description
01_raw	data/01_raw/	Parquet	Immutable source data.
02_intermediate	data/02_intermediate/	Parquet	Cleaned and feature-engineered data.
03_primary	data/03_primary/	Parquet	Model-ready splits (X_train, X_test, etc.).
06_models	data/06_models/	Pickle / JSON	Serialized models, metrics, and explanations.

​

Complete Dataset Registry

​

Raw Layer

Dataset	Type	Path
raw_bike_data	pandas.ParquetDataset	data/01_raw/raw_bike_data.parquet

​

Intermediate Layer

Dataset	Type	Path
preprocessed_bike_data	pandas.ParquetDataset	data/02_intermediate/preprocessed_bike_data.parquet
temporal_features	pandas.ParquetDataset	data/02_intermediate/temporal_features.parquet
engineered_features	pandas.ParquetDataset	data/02_intermediate/engineered_features.parquet
validated_features	pandas.ParquetDataset	data/02_intermediate/validated_features.parquet

​

Primary Layer

Dataset	Type	Path
model_input_table	pandas.ParquetDataSet	data/03_primary/model_input_table.parquet
X_train	pandas.ParquetDataSet	data/03_primary/X_train.parquet
X_test	pandas.ParquetDataSet	data/03_primary/X_test.parquet
y_train	pandas.ParquetDataSet	data/03_primary/y_train.parquet
y_test	pandas.ParquetDataSet	data/03_primary/y_test.parquet

​

Model Layer

Dataset	Type	Path
trained_model	pickle.PickleDataSet	data/06_models/trained_model.pkl
catboost_model	pickle.PickleDataset	data/06_models/catboost_model.pkl
regressor_model	pickle.PickleDataset	data/06_models/regressor_model.pkl
final_model	pickle.PickleDataset	data/06_models/final_model.pkl
model_metrics	json.JSONDataSet	data/06_models/model_metrics.json
feature_importance	json.JSONDataSet	data/06_models/feature_importance.json
shap_values	pickle.PickleDataSet	data/06_models/shap_values.pkl
prediction_explanation	json.JSONDataSet	data/06_models/prediction_explanation.json

​

Key Datasets

​

raw_bike_data

• Format: Parquet
• Schema (validated by RawBikeDataSchema):
  • datetime (String, nullable)
  • cnt (Target — total bike rental count)
  • temp, atemp, hum, windspeed (Normalized 0–1)
  • season, yr, mnth, hr, holiday, weekday, workingday, weathersit (Categorical integers)
  • casual, registered (Dropped during preprocessing)

​

model_input_table

• Format: Parquet
• Columns: All base features + lag features (cnt_lag_1, cnt_lag_2, cnt_lag_22, cnt_lag_23, hr_lag_1, etc.)
• Note: NaN values from lag creation are cleaned via backward fill then drop.

​

regressor_model

• Format: Pickle (CatBoost or RandomForest object)
• Versioning: Timestamped (e.g., regressor_20250101.pickle) and latest.

MLflow is the primary model registry. Catalog entries serve as fallback for local development.