Skip to main content

LlamaIndex

LlamaIndex talks to OpenAI-compatible endpoints out of the box, so any LlamaIndex pipeline runs against Ferro Labs AI Gateway by setting api_base on OpenAI. A dedicated llama-index-llms-ferrolabsai adapter is in development.

Drop-in (today)​

pip install llama-index llama-index-llms-openai
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="claude-3-5-sonnet-20241022",     # routed to Anthropic by Ferro
    api_base="http://localhost:8080/v1",
    api_key="sk-ferro-...",
)

print(llm.complete("Hello from LlamaIndex via Ferro").text)

Embeddings work the same way:

from llama_index.embeddings.openai import OpenAIEmbedding

embed = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base="http://localhost:8080/v1",
    api_key="sk-ferro-...",
)

Verify​

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-ferro-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

First-party adapter (planned)​

llama-index-llms-ferrolabsai 0.0.1 is a reserved PyPI placeholder. The 0.1.0 release will ship a real FerroLabsAI LLM class implementing LlamaIndex's LLM interface (complete, chat, stream_chat, achat), with trace_id, provider, and cost_usd surfaced on every response. It will later be upstreamed into run-llama/llama_index for LlamaIndex hub discoverability.

Track progress: Ferro AI Gateway roadmap.

Runnable example​

ai-gateway-cookbook/python/05-llamaindex-cheap-embeddings is planned alongside the adapter release.

See also​