Engineering
What this does: Details the microservices architecture — how the ML training layer, orchestration engine, and serving API are decoupled for independent scaling and deployment.
The system adopts a microservices-oriented architecture , explicitly decoupling the computation layer (Machine Learning) from the presentation layer (User Interface).
High-Level Design The architecture strictly separates the Kedro -based training pipelines, the Airflow orchestrator, and the FastAPI model serving layer.
Component Ecosystem Show 1. ML Service (app-ml)
The computational core powered by FastAPI and Kedro .
Role : Exposes high-performance REST endpoints for inference and training triggers.Pipeline Engine : Uses Kedro to orchestrate modular, reproducible data transformations and model training nodes.Model : CatBoost regressor optimized with Optuna .Port : 8000 (Inference), 8001 (Metrics) Show 2. Orchestration (Airflow)
The heartbeat of the MLOps lifecycle.
Role : Schedules and monitors recurring jobs (e.g., “Weekly Retraining”, “Daily Batch Inference”).Integration : Triggers Kedro nodes directly via the Airflow TaskFlow API , enabling granular observability.Observability : Provides a UI for tracking DAG success/failure and retrying halted pipelines. Show 3. Experiment Tracking (MLflow)
The centralized model registry and experiment logger.
Role : Tracks every training run, logging hyperparameters (learning_rate, depth), metrics (RMSE), and artifacts (model.pickle).Registry : Promotes the best performing models to “Staging” or “Production” for the API to consume. Show 4. UI Dashboard (app-ui)
The interactive presentation layer for business planners.
Tech Stack : Dash (Plotly) + Dash Bootstrap Components .Behavior : Stateless frontend that reads processed Parquet files directly from the shared volume/S3 for the latest forecasts.Port : 8050 Infrastructure
Shared Library (common) Enforces consistency by providing unified read_config and data_manager modules used by both training and inference.
PostgreSQL Persists forecast results, metadata, and Optuna study history for long-term storage.
