Skip to main content

Deploy with Docker Compose

Running an AI gateway alongside its logging database, cache layer, and monitoring stack as separate services is tedious and error-prone. Docker Compose lets you define the entire stack in a single file, wire everything together, and bring it up with one command.

This guide gives you a production-ready Compose file with the Ferro Labs AI Gateway, PostgreSQL (request logging), and Prometheus (metrics scraping). The gateway's built-in response-cache plugin is in-memory and runs inside the gateway process — there is no separate cache service to run, and distributed caching is not built in.

Prerequisites​

  • Docker Engine 20.10+ and Docker Compose v2
  • API keys for at least one upstream provider (OpenAI, Anthropic, etc.)

Project structure​

ferro-gateway/
├── docker-compose.yml
├── config.yaml
└── prometheus.yml

Gateway configuration​

Create config.yaml with your routing strategy, targets, and the request-logger, rate-limit, and response-cache plugins. Provider credentials are not placed in the config file — they are supplied to the container as environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.):

config.yaml
strategy:
  mode: fallback

targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
  - virtual_key: anthropic

# Map friendly names to actual model IDs.
aliases:
  fast: gpt-4o-mini
  smart: claude-sonnet-4-20250514

plugins:
  - name: request-logger
    type: logging
    stage: before_request
    enabled: true
    config:
      level: info
      # Persist request metadata to Postgres (the request-logger logs
      # metadata only — never prompt or response bodies).
      persist: true
      backend: postgres
      dsn: postgres://ferro:ferro_secret@postgres:5432/ferro_logs?sslmode=disable

  - name: rate-limit
    type: guardrail
    stage: before_request
    enabled: true
    config:
      requests_per_second: 120
      burst: 20

  # In-memory response cache (per-process; not distributed).
  - name: response-cache
    type: transform
    stage: before_request
    enabled: true
    config:
      max_age: 3600
      max_entries: 1000

Docker Compose file​

docker-compose.yml
version: "3.9"

services:
  gateway:
    image: ghcr.io/ferro-labs/ai-gateway:latest
    container_name: ferro-gateway
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - GATEWAY_CONFIG=/etc/ferro/config.yaml
      - PORT=8080
    volumes:
      - ./config.yaml:/etc/ferro/config.yaml:ro
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: postgres:16-alpine
    container_name: ferro-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=ferro
      - POSTGRES_PASSWORD=ferro_secret
      - POSTGRES_DB=ferro_logs
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ferro -d ferro_logs"]
      interval: 5s
      timeout: 3s
      retries: 5

  prometheus:
    image: prom/prometheus:v2.53.0
    container_name: ferro-prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    depends_on:
      gateway:
        condition: service_healthy

volumes:
  postgres_data:
  prometheus_data:

Volume mounts​

ServiceMountPurpose
gateway./config.yaml (read-only)Gateway strategy, targets, and plugin config
postgrespostgres_data named volumePersist request logs across restarts
prometheus./prometheus.yml (read-only)Prometheus scrape configuration
prometheusprometheus_data named volumePersist metrics data across restarts

Environment variables​

Create a .env file in the same directory as docker-compose.yml:

.env
OPENAI_API_KEY=sk-proj-your-openai-key-here
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here

Docker Compose automatically reads .env and injects the values into the ${...} placeholders in the Compose file.

warning

Never commit .env to version control. Add it to your .gitignore:

echo ".env" >> .gitignore

Prometheus scrape configuration​

Create prometheus.yml to scrape the gateway's /metrics endpoint:

prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "ferro-gateway"
    metrics_path: /metrics
    static_configs:
      - targets: ["gateway:8080"]
        labels:
          environment: "production"
          service: "ai-gateway"

Health check configuration​

The gateway exposes a /health endpoint that returns 200 OK when the service is ready. The Compose file configures Docker to:

  1. Probe the endpoint every 10 seconds with a 5-second timeout.
  2. Retry up to 3 times before marking the container unhealthy.
  3. Wait 5 seconds after container start before the first probe (start_period).

Dependent services (Prometheus) only start after the gateway is healthy.

Test it​

Bring up the entire stack and verify:

docker compose up -d

Wait a few seconds for the health checks, then:

curl http://localhost:8080/health

You should see:

{"status":"healthy"}

Send a test request through the gateway:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello from Docker Compose!"}]
  }'

Check Prometheus targets at http://localhost:9090/targets to confirm the gateway is being scraped.

tip

To view logs across all services in real time:

docker compose logs -f

To tear down the stack while preserving volumes:

docker compose down

To tear down and delete all data:

docker compose down -v