Inference & Serving

FastAPI prediction API with Argon2 auth, rate limiting, A/B testing, and Prometheus observability

Engineering

What this does: Generates real-time demand predictions via a secure, rate-limited API — enabling business applications to request forecasts on demand with enterprise-grade authentication and monitoring.

The system provides a robust Real-Time Inference API powered by FastAPI with production-grade security, observability, and traffic management.

​

Architecture

The serving layer (src/demand_forecasting/serving) exposes a REST API for demand forecasting.

​

Security Middleware Stack

The API applies four layers of middleware, in order of priority:

1. SecurityHeadersMiddleware

   Adds OWASP-recommended headers: X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Content-Security-Policy, Referrer-Policy, Permissions-Policy. In production, adds Strict-Transport-Security (HSTS).

2. RateLimitMiddleware

   IP-based rate limiting (default: 60 requests/minute). Returns 429 Too Many Requests with Retry-After and X-RateLimit-* headers. Configurable via RATE_LIMIT_PER_MINUTE environment variable.

3. TrustedHostMiddleware

   Prevents Host header attacks. Allowed hosts are environment-specific — production hosts must be explicitly configured via ALLOWED_HOSTS.

4. CORS

   Environment-aware origin allowlisting. Only GET, POST, OPTIONS methods allowed. No wildcard origins in any environment.

​

Authentication

API key authentication using Argon2id hashing with constant-time verification:

• Key Storage: SHA-256 → Argon2id hashed in memory (loaded from VALID_API_KEYS env var)
• Brute Force Protection: Max 10 failed attempts per hour per IP
• Rehashing: Automatic rehash if Argon2 parameters are updated

All /predict and /predict/ab endpoints require the X-API-Key header. Unauthenticated requests receive a 401 Unauthorized response.

​

Inference Strategies

Show A/B Testing & Traffic Splitting

Simultaneously serve two models to compare real-world performance.

• Weighted Routing: Configurable split (e.g., AB_SPLIT=0.7).
• Per-Request Routing: Each request is independently routed based on the split ratio.
• Enable: Set ENABLE_AB_TESTING=true and MODEL_B_URI environment variables.

Show Shadow Mode (Challenger)

New models receive live traffic for validation without returning results to the user.

• Safe Validation: Calculate metrics on live distributions before promotion.

Show Prediction Confidence

API returns a Confidence Score (0.50 to 0.95).

• Default: 0.75 for standard regression predictions.
• CatBoost: Uses leaf index variance as uncertainty proxy when available.

​

API Reference

POST /predict

Generates a forecast for a given set of conditions. Requires authentication.

Request

import requests
response = requests.post(
    "http://localhost:8000/predict",
    headers={"X-API-Key": "your-api-key"},
    json={
        "features": {
            "hr": 14, "temp": 0.5, "hum": 0.6,
            "weathersit": 1, "weekday": 2
        }
    }
)

GET /health

Verifies service health. Returns 200 OK with model count and timestamp. No auth required.

Response

{
  "status": "healthy",
  "models_loaded": 1,
  "timestamp": "2025-01-15T14:30:00"
}

GET /model/info

Returns current model metadata including type, features, and version. No auth required.

Response

{
  "model_version": "production",
  "model_type": "CatBoost",
  "features": ["temp", "hum", "windspeed", "hr", "weekday"],
  "last_updated": "2025-01-15T14:30:00",
  "model_id": "production"
}

GET /metrics/drift

Returns Evidently-powered drift metrics with 5-minute caching. No auth required.

Response

{
  "status": "ok",
  "cached": false,
  "last_check": "2025-01-15T14:30:00",
  "metrics": {
    "evidently_dataset_drift": 0.0,
    "evidently_drift_share": 0.12,
    "evidently_columns_total": 10,
    "evidently_columns_drifted": 1
  }
}

GET /metrics

Exposes Prometheus metrics:

• prediction_requests_total (labels: model_id, status)
• prediction_duration_seconds (histogram)
• model_accuracy (gauge per model_id)
• evidently_dataset_drift / evidently_drift_share (gauges)

---

Refer to the API Reference for complete schema details.

​

Related Pages

Generate Forecast API

Full API reference with request/response schemas and interactive playground.

Infrastructure

Docker Compose architecture and service configuration behind the API.