> ## Documentation Index
> Fetch the complete documentation index at: https://freelance-4888ed08.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common errors, diagnostic queries, and resolution guides.

## Overview

This guide catalogs common issues encountered in the `demand_forecasting` system, particularly related to the MinIO/PostgreSQL infrastructure and Airflow DAG execution.

```mermaid theme={null}
graph TD
    %% Styling
    classDef start fill:#E0E7FF,stroke:#4338CA,color:#1E1B4B,stroke-width:2px;
    classDef db fill:#DCFCE7,stroke:#15803D,color:#14532D,stroke-width:2px;
    classDef python fill:#DBEAFE,stroke:#2563EB,color:#1E3A5F,stroke-width:2px;
    classDef network fill:#FEF3C7,stroke:#B45309,color:#78350F,stroke-width:2px;
    classDef dag fill:#F3E8FF,stroke:#7C3AED,color:#2E1065,stroke-width:2px;
    classDef fix fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:2px;

    Issue["Service Error<br/><i>What failed?</i>"]:::start
    Issue --> DBErr{"Too many<br/>connections?"}:::db
    Issue --> PyErr{"Module not<br/>found?"}:::python
    Issue --> NetErr{"Connection<br/>refused?"}:::network
    Issue --> DagErr{"DAG search<br/>space failed?"}:::dag

    DBErr -->|Yes| FixDB["Increase max_connections<br/><i>or reduce pool_size</i>"]:::fix
    PyErr -->|Yes| FixPy["Set PYTHONPATH<br/><i>/opt/airflow/src</i>"]:::fix
    NetErr -->|Yes| FixNet["Use Docker DNS<br/><i>minio:9000</i>"]:::fix
    DagErr -->|Yes| FixDag["Check MinIO<br/><i>Bucket Permissions</i>"]:::fix

    %% Link Styling
    linkStyle default stroke:#6366f1,stroke-width:2px;
```

## Common Issues & Solutions

### 1. "Too many connections" (PostgreSQL)

**Symptom**: Airflow tasks fail with `FATAL: sorry, too many clients already`.

**Root Cause**: The parallel optimization (Optuna) combined with Airflow's own database usage has exceeded the `max_connections` limit.

**Fix**:

1. Check effective connections:
   ```sql theme={null}
   SELECT count(*) FROM pg_stat_activity;
   ```
2. Increase PostgreSQL limit in `postgresql.conf`:
   ```ini theme={null}
   max_connections = 200  # Default is often 100
   ```
3. Restart `postgres` container.

### 2. "ModuleNotFoundError: No module named 'demand\_forecasting'"

**Symptom**: Airflow DAGs fail to import.

**Root Cause**: The `PYTHONPATH` has not been set correctly in the Docker environment.

**Fix**:
Ensure your `.env` file or `docker-compose.yaml` includes:

```yaml theme={null}
environment:
  PYTHONPATH: /opt/airflow/src
```

### 3. "Connection Refused" (MinIO)

**Symptom**: `EndpointConnectionError` when tasks try to read/write S3.

**Root Cause**: Airflow cannot resolve `localhost` because it runs inside Docker. It must use the internal Docker network alias.

**Fix**:
Use `http://minio:9000` (internal service name) instead of `http://localhost:9000` in your `credentials.yml` or environment variables.

### 4. DAG "Search Space Generation Failed"

**Symptom**: `ValueError: No trial files found`.

**Root Cause**: The `t_generate_search_space` task successfully generated parameters, but failed to write the JSON files to MinIO, or the training task is looking in the wrong bucket path.

**Fix**:

* Verify MinIO bucket permissions.
* Check the mapped volume for `minio` in docker-compose.

## Diagnostic Queries

### Check Optuna Study State

```sql theme={null}
SELECT study_name, s.study_id, count(t.trial_id) as trials 
FROM studies s 
JOIN trials t ON s.study_id = t.study_id 
GROUP BY 1, 2;
```

### Check Drift History

```sql theme={null}
SELECT feature_name, drift_score, check_timestamp 
FROM drift_records 
WHERE drift_detected = true 
ORDER BY check_timestamp DESC 
LIMIT 10;
```

## Logs Location

* **Airflow Scheduler**: `docker-compose logs -f airflow-scheduler` (DAG parsing issues)
* **Airflow Worker**: `docker-compose logs -f airflow-worker` (Task execution errors)
* **PostgreSQL**: `docker-compose logs -f postgres` (DB connection issues)
* **Model Server**: `docker-compose logs -f model-server` (Inference errors)
* **MinIO**: `docker-compose logs -f minio` (Storage access issues)

## Additional Scenarios

> [accordion] model-server container keeps restarting
>
> **Symptom**: `docker-compose ps` shows `model-server` in a restart loop; logs show `SyntaxError` or `ModuleNotFoundError`.
>
> **Root Cause**: Python dependencies out of sync with the codebase, or a syntax error introduced in a recent commit.
>
> **Fix**:
>
> 1. Check logs: `docker-compose logs --tail=50 model-server`
> 2. Rebuild: `docker-compose build model-server --no-cache`
> 3. Ensure `requirements.txt` includes all imports from `app-ml/`
> 4. Restart: `docker-compose up -d model-server`

> [accordion] MLflow UI shows no experiments
>
> **Symptom**: `http://localhost:5000` loads but the experiment list is empty.
>
> **Root Cause**: Tracking URI mismatch or no training run has been executed yet.
>
> **Fix**:
>
> 1. Verify `.env` contains: `MLFLOW_TRACKING_URI=http://mlflow:5000`
> 2. Run at least one training DAG to populate experiments
> 3. Check the MLflow backend store: `docker-compose exec mlflow ls /mlflow/mlruns/`

> [accordion] Drift metrics return empty
>
> **Symptom**: `/metrics/drift` returns `{"metrics": {}}`.
>
> **Root Cause**: No reference data available, or insufficient predictions have been collected.
>
> **Fix**:
>
> 1. Ensure reference data exists at `data/01_raw/raw_bike_data.parquet`
> 2. Confirm at least 50 predictions have been made via `/predict`
> 3. Verify `evidently` is installed in the model-server container

> [accordion] Airflow DAGs not appearing in UI
>
> **Symptom**: Airflow UI at `:8081` loads but the DAG list is empty.
>
> **Root Cause**: DAG files not mounted correctly, or import errors preventing DAG discovery.
>
> **Fix**:
>
> 1. Verify the volume mount in `docker-compose.yml`: `./dags:/opt/airflow/dags`
> 2. Check imports: `docker-compose exec airflow-scheduler python -c "import demand_forecasting"`
> 3. Review scheduler logs: `docker-compose logs -f airflow-scheduler | grep ERROR`
> 4. Confirm `AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags` in `.env`

## Docker Quick Commands

```bash theme={null}
# Restart all services
docker-compose down && docker-compose up -d

# Rebuild a specific service
docker-compose build <service-name> --no-cache

# View real-time logs for a service
docker-compose logs -f <service-name>

# Check service health
docker-compose ps

# Access MinIO console
# Open http://localhost:9001 (default: minioadmin/minioadmin)

# Access Airflow UI
# Open http://localhost:8081 (default: airflow/airflow)
```

***

## Related Pages

> [card] [Infrastructure](https://freelance-4888ed08.mintlify.site/deployment/infrastructure.md)
>
> Full service architecture and Docker Compose configuration.

> [card] [Environment Variables](https://freelance-4888ed08.mintlify.site/configuration/env-vars.md)
>
> All environment variables referenced in troubleshooting fixes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.