> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Catalog

> Dataset Definitions and Storage Formats

The Data Catalog (`conf/base/catalog.yml`) serves as the single source of truth for all data assets in the system.

## Data Layers

We follow the Kedro data engineering convention:

| Layer | Path Pattern | Format | Description |
| :- | :- | :- | :- |
| **01\_raw** | `data/01_raw/` | Parquet | Immutable source data. |
| **02\_intermediate** | `data/02_intermediate/` | Parquet | Cleaned and feature-engineered data. |
| **03\_primary** | `data/03_primary/` | Parquet | Model-ready splits (X\_train, X\_test, etc.). |
| **06\_models** | `data/06_models/` | Pickle / JSON | Serialized models, metrics, and explanations. |

## Complete Dataset Registry

### Raw Layer

| Dataset | Type | Path |
| :- | :- | :- |
| `raw_bike_data` | `pandas.ParquetDataset` | `data/01_raw/raw_bike_data.parquet` |

### Intermediate Layer

| Dataset | Type | Path |
| :- | :- | :- |
| `preprocessed_bike_data` | `pandas.ParquetDataset` | `data/02_intermediate/preprocessed_bike_data.parquet` |
| `temporal_features` | `pandas.ParquetDataset` | `data/02_intermediate/temporal_features.parquet` |
| `engineered_features` | `pandas.ParquetDataset` | `data/02_intermediate/engineered_features.parquet` |
| `validated_features` | `pandas.ParquetDataset` | `data/02_intermediate/validated_features.parquet` |

### Primary Layer

| Dataset | Type | Path |
| :- | :- | :- |
| `model_input_table` | `pandas.ParquetDataSet` | `data/03_primary/model_input_table.parquet` |
| `X_train` | `pandas.ParquetDataSet` | `data/03_primary/X_train.parquet` |
| `X_test` | `pandas.ParquetDataSet` | `data/03_primary/X_test.parquet` |
| `y_train` | `pandas.ParquetDataSet` | `data/03_primary/y_train.parquet` |
| `y_test` | `pandas.ParquetDataSet` | `data/03_primary/y_test.parquet` |

### Model Layer

| Dataset | Type | Path |
| :- | :- | :- |
| `trained_model` | `pickle.PickleDataSet` | `data/06_models/trained_model.pkl` |
| `catboost_model` | `pickle.PickleDataset` | `data/06_models/catboost_model.pkl` |
| `regressor_model` | `pickle.PickleDataset` | `data/06_models/regressor_model.pkl` |
| `final_model` | `pickle.PickleDataset` | `data/06_models/final_model.pkl` |
| `model_metrics` | `json.JSONDataSet` | `data/06_models/model_metrics.json` |
| `feature_importance` | `json.JSONDataSet` | `data/06_models/feature_importance.json` |
| `shap_values` | `pickle.PickleDataSet` | `data/06_models/shap_values.pkl` |
| `prediction_explanation` | `json.JSONDataSet` | `data/06_models/prediction_explanation.json` |

## Key Datasets

### `raw_bike_data`

* **Format**: Parquet
* **Schema** (validated by `RawBikeDataSchema`):
  * `datetime` (String, nullable)
  * `cnt` (Target — total bike rental count)
  * `temp`, `atemp`, `hum`, `windspeed` (Normalized 0–1)
  * `season`, `yr`, `mnth`, `hr`, `holiday`, `weekday`, `workingday`, `weathersit` (Categorical integers)
  * `casual`, `registered` (Dropped during preprocessing)

### `model_input_table`

* **Format**: Parquet
* **Columns**: All base features + lag features (`cnt_lag_1`, `cnt_lag_2`, `cnt_lag_22`, `cnt_lag_23`, `hr_lag_1`, etc.)
* **Note**: `NaN` values from lag creation are cleaned via backward fill then drop.

### `regressor_model`

* **Format**: Pickle (`CatBoost` or `RandomForest` object)
* **Versioning**: Timestamped (e.g., `regressor_20250101.pickle`) and `latest`.

> [!NOTE]
>
> MLflow is the primary model registry. Catalog entries serve as fallback for local development.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.