Troubleshooting

Common errors, diagnostic queries, and resolution guides.

​

Overview

This guide catalogs common issues encountered in the demand_forecasting system, particularly related to the MinIO/PostgreSQL infrastructure and Airflow DAG execution.

​

Common Issues & Solutions

​

1. “Too many connections” (PostgreSQL)

Symptom: Airflow tasks fail with FATAL: sorry, too many clients already.

Root Cause: The parallel optimization (Optuna) combined with Airflow’s own database usage has exceeded the max_connections limit.

Fix:

1. Check effective connections:
   SELECT count(*) FROM pg_stat_activity;
   
2. Increase PostgreSQL limit in postgresql.conf:
   max_connections = 200  # Default is often 100
   
3. Restart postgres container.

​

2. “ModuleNotFoundError: No module named ‘demand_forecasting’”

Symptom: Airflow DAGs fail to import.

Root Cause: The PYTHONPATH has not been set correctly in the Docker environment.

Fix: Ensure your .env file or docker-compose.yaml includes:

environment:
  PYTHONPATH: /opt/airflow/src

​

3. “Connection Refused” (MinIO)

Symptom: EndpointConnectionError when tasks try to read/write S3.

Root Cause: Airflow cannot resolve localhost because it runs inside Docker. It must use the internal Docker network alias.

Fix: Use http://minio:9000 (internal service name) instead of http://localhost:9000 in your credentials.yml or environment variables.

​

4. DAG “Search Space Generation Failed”

Symptom: ValueError: No trial files found.

Root Cause: The t_generate_search_space task successfully generated parameters, but failed to write the JSON files to MinIO, or the training task is looking in the wrong bucket path.

Fix:

• Verify MinIO bucket permissions.
• Check the mapped volume for minio in docker-compose.

​

Diagnostic Queries

​

Check Optuna Study State

SELECT study_name, s.study_id, count(t.trial_id) as trials 
FROM studies s 
JOIN trials t ON s.study_id = t.study_id 
GROUP BY 1, 2;

​

Check Drift History

SELECT feature_name, drift_score, check_timestamp 
FROM drift_records 
WHERE drift_detected = true 
ORDER BY check_timestamp DESC 
LIMIT 10;

​

Logs Location

• Airflow Scheduler: docker-compose logs -f airflow-scheduler (DAG parsing issues)
• Airflow Worker: docker-compose logs -f airflow-worker (Task execution errors)
• PostgreSQL: docker-compose logs -f postgres (DB connection issues)
• Model Server: docker-compose logs -f model-server (Inference errors)
• MinIO: docker-compose logs -f minio (Storage access issues)

​

Additional Scenarios

model-server container keeps restarting

Symptom: docker-compose ps shows model-server in a restart loop; logs show SyntaxError or ModuleNotFoundError.

Root Cause: Python dependencies out of sync with the codebase, or a syntax error introduced in a recent commit.

Fix:

1. Check logs: docker-compose logs --tail=50 model-server
2. Rebuild: docker-compose build model-server --no-cache
3. Ensure requirements.txt includes all imports from app-ml/
4. Restart: docker-compose up -d model-server

MLflow UI shows no experiments

Symptom: http://localhost:5000 loads but the experiment list is empty.

Root Cause: Tracking URI mismatch or no training run has been executed yet.

Fix:

1. Verify .env contains: MLFLOW_TRACKING_URI=http://mlflow:5000
2. Run at least one training DAG to populate experiments
3. Check the MLflow backend store: docker-compose exec mlflow ls /mlflow/mlruns/

Drift metrics return empty

Symptom: /metrics/drift returns {"metrics": {}}.

Root Cause: No reference data available, or insufficient predictions have been collected.

Fix:

1. Ensure reference data exists at data/01_raw/raw_bike_data.parquet
2. Confirm at least 50 predictions have been made via /predict
3. Verify evidently is installed in the model-server container

Airflow DAGs not appearing in UI

Symptom: Airflow UI at :8081 loads but the DAG list is empty.

Root Cause: DAG files not mounted correctly, or import errors preventing DAG discovery.

Fix:

1. Verify the volume mount in docker-compose.yml: ./dags:/opt/airflow/dags
2. Check imports: docker-compose exec airflow-scheduler python -c "import demand_forecasting"
3. Review scheduler logs: docker-compose logs -f airflow-scheduler | grep ERROR
4. Confirm AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags in .env

​

Docker Quick Commands

# Restart all services
docker-compose down && docker-compose up -d

# Rebuild a specific service
docker-compose build <service-name> --no-cache

# View real-time logs for a service
docker-compose logs -f <service-name>

# Check service health
docker-compose ps

# Access MinIO console
# Open http://localhost:9001 (default: minioadmin/minioadmin)

# Access Airflow UI
# Open http://localhost:8081 (default: airflow/airflow)

---

​

Related Pages

Infrastructure

Full service architecture and Docker Compose configuration.

Environment Variables

All environment variables referenced in troubleshooting fixes.