Deployment Strategies

Safe rollout procedures from Staging to Production

OperationsEngineering

What this does: Defines how new model versions move safely from staging to production using a 2-week promotion cycle — ensuring zero-downtime updates and easy rollback if issues arise.

Our deployment strategy prioritizes safety and zero-downtime updates. We utilize a 2-week promotion cycle with distinct “Staging” and “Production” environments detached via Docker Compose configuration.

​

Environment Architecture

Env	Docker File	Purpose	Data Source
Dev	docker-compose.yml	Feature development	Local / MinIO (Dev)
Staging	docker-compose.staging.yml	Integration testing	MinIO (Staging Replica)
Prod	docker-compose.prod.yml	Live traffic	MinIO (Production)

​

Promotion Lifecycle

​

Week 1: Staging (Integration)

1. Deploy Candidate: Tag the release v3.0.0-rc1.
2. Smoke Tests: Run critical path tests (pytest -m smoke).
3. Performance Verification: Ensure inference latency < 200ms.

# Example: Deploying to Staging
export AIRFLOW_ENV=staging
docker-compose -f docker-compose.staging.yml up -d --build
./scripts/health_check.sh

​

Week 2: Production (Go Live)

1. Backup: Snapshot the current PostgreSQL metadata.
2. Traffic Shift: In a Blue/Green setup, route traffic to the new containers.
3. Monitoring: Watch DriftDetector logs for immediate anomalies.

​

Scripts & Automation

We use tailored scripts to automate these risky operations.

Show Backup Script

scripts/backup_production.sh

Creates a timestamped dump of the Airflow metadata DB and Optuna studies.

pg_dump -h postgres -U airflow > backup_$(date +%F).sql

Show Health Check

scripts/verify_ml_integration.py

A comprehensive integrity check that validates PostgreSQL, MinIO, and MLflow connections before traffic is shifted.

​

Rollback Procedure

If a critical failure is detected (e.g., Error Rate > 1%), execute an immediate rollback:

1. Revert Image: Point docker-compose.prod.yml back to v2.x.x.
2. Restore DB: If schema migration failed, restore from the pre-deploy backup.
3. Restart: docker-compose up -d --force-recreate.