Experiments & Lab

From Jupyter Notebook prototypes to Production Pipelines

Data Science

What this does: Provides a laboratory environment for rapid prototyping and exploratory data analysis — code developed here graduates into the production pipeline, with full experiment tracking via MLflow.

The “Lab” environment consists of Jupyter Notebooks used for Exploratory Data Analysis (EDA) and rapid prototyping. Code developed here is the precursor to the robust src/ pipelines.

​

Notebook Inventory

Located in app-ml/notebooks/:

Notebook	Purpose	Key Outputs
EDA.ipynb	Initial data investigation	Correlation matrices, Seasonality plots, Missing value analysis
Modeling.ipynb	Algorithms prototyping	Baseline Random Forest performance, CatBoost feature importance

​

Workflow: Lab to Production

We follow a strict promotion process to ensure experimental code is safely productionized.

​

1. Exploratory Data Analysis (EDA.ipynb)

Data Scientists use this notebook to understand the dataset distribution. Key insights derived here informed the pipeline design:

• Seasonality: Confirmed strong daily and weekly cycles, leading to the creation of lag features [1, 2, 22, 23].
• Correlations: Identified temp and atemp as highly collinear, leading to the decision to use only temp.

​

2. Modeling Prototypes (Modeling.ipynb)

Used to compare algorithms before settling on CatBoost.

• Tested: Linear Regression, Random Forest, XGBoost, CatBoost.
• Result: CatBoost showed superior handling of categorical variables (season, weathersit) without extensive one-hot encoding.

​

Running Notebooks Locally

To run the notebooks within the correct environment:

1. Activate Environment

   micromamba activate demand-forecasting
   
2. Launch Jupyter

   cd app-ml/notebooks
   jupyter lab
   
3. Kernel Selection

   Ensure the kernel is set to demand-forecasting to access project dependencies like kedro, optuna, and pandas.