Infrastructure

Docker Compose architecture with 20 services including MLflow, Airflow, Prometheus, and FastAPI model serving

Engineering

What this does: Runs the entire forecasting platform as 20 containerized services with a single command — providing enterprise-grade reliability, monitoring, and scaling without complex server management.

​

Containerization Strategy

The entire system is containerized using Docker, with Docker Compose orchestrating a 20-service multi-container environment across 5 functional groups.

​

Service Groups

Shared Infrastructure

Service	Image	Port(s)	Purpose
postgres	postgres:15.1-alpine	5432	Shared DB (Airflow, MLflow, Optuna)
redis	redis:7-alpine	6379	Caching & Celery broker
minio	minio/minio:latest	9000, 9001	S3-compatible artifact store
createbuckets	minio/mc:latest	—	Initializes MinIO buckets on startup

ML & Orchestration

Service	Image	Port(s)	Purpose
mlflow	python:3.11-slim	5000	Experiment tracking & model registry
airflow-webserver	Custom build	8081	DAG management UI
airflow-scheduler	Custom build	—	Schedules & executes tasks
airflow-triggerer	Custom build	—	Async trigger processing
airflow-init	Custom build	—	DB migration & admin creation
airflow-worker	Custom build	—	Celery worker (disabled by default)

Model Serving

Service	Image	Port(s)	Purpose
model-server	Dockerfile.model-server.ultra-optimized	8000, 8001	FastAPI inference + metrics
nginx	nginx:alpine	80, 8080	Load balancer & reverse proxy
test-runner	Same as model-server	—	pytest (testing profile only)

Monitoring

Service	Image	Port(s)	Purpose
prometheus	prom/prometheus:latest	9090	Metrics collection (15d retention, 10GB limit)
grafana	grafana/grafana:latest	3000	Dashboards & alerting
node-exporter	prom/node-exporter:latest	9100	Host CPU/memory metrics

Experiment Tracking

Service	Image	Port(s)	Purpose
clearml-apiserver	clearml/server:latest	8008	ClearML experiment API
clearml-webserver	clearml/server:latest	8082	ClearML web UI
clearml-fileserver	clearml/server:latest	8083	ClearML file storage
clearml-mongo	mongo:4.4	—	ClearML metadata store
clearml-elastic	elasticsearch:7.17.0	—	ClearML search index
optuna-dashboard	ghcr.io/optuna/optuna-dashboard:latest	8084	HPO visualization
app-ui	Custom build (../app-ui)	8050	Dash analytics frontend

​

Networking

All containers share a single bridge network defined in docker-compose.yml:

docker-compose.yml

networks:
  mlops-network:
    driver: bridge
    name: demand-forecasting-network

Services communicate using Docker DNS (e.g., the UI calls http://model-server:8000).

​

Resource Constraints

All services have CPU and memory limits defined to prevent resource contention:

Service	CPU Limit	Memory Limit	CPU Reserved	Memory Reserved
postgres	1.0	1 GB	0.5	512 MB
minio	1.0	1 GB	0.5	512 MB
model-server	1.0	2 GB	0.5	1 GB
redis	0.5	512 MB	0.25	256 MB
mlflow	0.5	1 GB	0.25	512 MB

​

Environment Variables

Key environment variables (set via .env file):

Variable	Purpose
POSTGRES_PASSWORD	Shared PostgreSQL password
REDIS_PASSWORD	Redis authentication
MINIO_ROOT_USER / MINIO_ROOT_PASSWORD	MinIO admin credentials
AIRFLOW__CORE__FERNET_KEY	Airflow encryption key
MODEL_SERVER_URL	URL where the UI finds the ML service
VALID_API_KEYS	Comma-separated API keys for model server auth

​

Optimization

We use a Multi-Stage Build strategy in Dockerfile.model-server.ultra-optimized to ensure fast builds and small images:

1. Cache Mounts: Uses --mount=type=cache,target=/root/.cache/pip to speed up dependency installation by 40–50%.
2. Layer Separation: Copies requirements files before source code to maximize Docker layer caching.
3. Non-Root User: Runs as appuser for security compliance.

​

MinIO Buckets

The createbuckets init container creates the following S3 buckets on first startup:

Bucket	Purpose
demand-forecasting	Raw data, features, model artifacts
mlflow-artifacts	MLflow experiment artifacts
airflow-logs	Remote Airflow task logs
drift-reports	Evidently drift analysis reports
clearml-artifacts	ClearML experiment data

---

​

Related Pages

Environment Variables

Complete reference for all environment variables used across services.

Inference & Serving

FastAPI model serving layer that runs inside the model-server container.