Skip to main content

Performance Benchmarks

Go-native means fast. Here's the proof.

Every number on this page comes from reproducible, open-source benchmarks. No synthetic micro-benchmarks — real gateway overhead measured under sustained load against a mock upstream with constant latency.

Methodology​

Test Environment & Parameters

Hardware: Dedicated Linux VM — 4 vCPU, 8 GB RAM

Load generator: k6 (Grafana)

Upstream: Mock server returning a fixed response after a constant 50 ms latency. This isolates gateway overhead from provider variance.

RPS levels tested: 100, 200, 500, 1 000, 2 000

Duration: 60 seconds sustained at each level

What was measured: Gateway-added overhead only (total response time minus the 50 ms upstream latency). Memory sampled via docker stats at 1 s intervals.

All tests run five times; tables report median of medians (p50) and median of p99s.

Gateway version and config not recorded

The original benchmark report does not state which gateway version or runtime configuration produced the numbers below, and doesn't say whether auth (ALLOW_UNAUTHENTICATED_PROXY), the per-IP rate limiter (RATE_LIMIT_RPS, on by default since v1.1.18 at 20 rps / burst 40), or the request-logger plugin were enabled during the run. All three sit on the request hot path and add measurable overhead, and AI Gateway's current shipping defaults — auth required on /v1/*, per-IP limiting on, plus the v1.4.0 unified routing pipeline — may not match what was measured here.

Treat these numbers as directional rather than a v1.4.x guarantee. Reproduce the benchmarks against the version and config you plan to run to get numbers you can cite.

Ferro Labs AI Gateway vs Python-Based Alternatives at 500 RPS​

Head-to-head at 500 requests per second sustained for 60 seconds. This is the threshold where Python-based gateways begin showing consistent p99 degradation beyond 200 RPS.

MetricAI Gateway (Go)Python-based gateways
p50 overhead<0.5 ms~3 ms
p99 overhead<1 ms~15 ms+
Throughput500 RPS sustainedDegraded (drops under target)
Memory~120 MB~400 MB+
Success rate100%<99.5%

The Go AI gateway latency advantage compounds at scale. At 500 RPS, the p99 gap is over 15x — and Python alternatives start dropping requests entirely.

Scaling Behavior: AI Gateway Across RPS Levels​

The fastest AI gateway should add near-zero overhead regardless of load. Here is how AI Gateway scales from 100 to 2 000 RPS:

RPSp50 overheadp99 overheadThroughputMemorySuccess rate
1000.2 ms0.4 ms100 RPS sustained~45 MB100%
2000.3 ms0.5 ms200 RPS sustained~60 MB100%
5000.4 ms0.8 ms500 RPS sustained~120 MB100%
1 0000.5 ms0.9 ms1 000 RPS sustained~180 MB100%
2 0000.6 ms1.1 ms2 000 RPS sustained~250 MB100%

Sub-millisecond p99 overhead holds through 1 000 RPS. Even at 2 000 RPS the gateway adds just over 1 ms at the tail — well within noise for any LLM API call that takes 200 ms–2 s on the provider side.

Why Go?​

Why a Go AI gateway outperforms Python-based LLM proxies
  • Goroutines for concurrency — thousands of in-flight requests multiplexed onto a small thread pool with near-zero scheduling cost. No thread-per-request overhead.
  • No GIL — every CPU core does real work in parallel. Python's Global Interpreter Lock serializes CPU-bound gateway logic (auth, routing, logging) across all requests.
  • Low-GC overhead with careful allocation — arena-style buffering and sync.Pool reuse keep heap churn minimal. GC pauses stay under 0.5 ms even at 2 000 RPS.
  • Single static binary — no interpreter, no virtualenv, no pip install at deploy time. One COPY in your Dockerfile. (Not zero-dependency at runtime: the gateway fetches the model catalog on startup by default — set FERRO_MODEL_CATALOG_TIMEOUT=0 for air-gapped deploys — and optional stores like Postgres are runtime dependencies you configure, not compiled in.)

Reproduce the Benchmarks​

The full benchmark suite is open source. Clone it, run it, verify every number on this page. This is a LiteLLM alternative performance comparison you can audit yourself — pin the gateway image to the version you care about (e.g. ghcr.io/ferro-labs/ai-gateway:v1.4.1) since, as noted above, these numbers have not been re-measured against the current release.

# Clone the benchmark repository
git clone https://github.com/ferro-labs/ai-gateway-performance-benchmarks.git
cd ai-gateway-performance-benchmarks
# Start the mock upstream and gateway containers
docker compose up -d
# Run the full benchmark suite (100–2000 RPS, 60s each)
./run-benchmarks.sh
# Or run a single RPS level
k6 run --env RPS=500 --env DURATION=60s scripts/gateway-overhead.js

Results are written to results/ as JSON. The included plot.py script generates comparison charts.

# Generate comparison plots
python3 plot.py results/