The LLM Gateway is Nova’s model routing layer. It exposes a unified API that translates requests to any configured provider — Anthropic, OpenAI, Ollama, Groq, Gemini, Cerebras, OpenRouter, GitHub Models, and the ChatGPT Plus subscription provider.
Requests can carry tier: best | mid | cheap instead of a concrete model
(pods and agents can pin tier:<name> the same way). The resolver walks the
tier’s preference list and picks the first candidate that passes validated
discovery — the provider’s credential answered a real API call and the model
is on its current live list — plus quota and context-window checks. Dead keys,
retired models, and un-pulled local models are skipped, so a tier hint can
never resolve to a model that doesn’t exist. An empty tier falls through to
the next cheaper tier. Preference lists are seeded from
TIER_PREFERENCES_BEST/MID/CHEAP and overridable at runtime via the Redis
config key llm.tier_preferences (JSON object of tier → model list).
GET /v1/models/tiers shows every candidate’s verdict and what each tier
resolves to right now.
A provider whose API key is rejected (expired, revoked, or never configured)
never takes a request down with it:
The failing provider is sidelined for a 10-minute cooldown and skipped by
the fallback chain; the request is retried once on the local default model
(unless the strategy is cloud-only, which returns a structured
502 provider_credentials_invalid instead of a raw error).
Paid providers are isolated per credential — a dead Anthropic key never
sidelines OpenAI, and vice versa.
GET /health/providers reports credential_invalid: true for any provider
currently in cooldown, so a bad key is visible at a glance. Rotate keys in
Settings → AI & Models → Provider Status — the new key applies live
(no restart): the gateway hot-reloads its credentials the moment a key is
saved or removed, rebuilds its failover chains, and clears the cooldown so
a fixed key is retried immediately.
Wrapper that reads active backend config from Redis (5s cache) and delegates to the appropriate provider (Ollama, vLLM, SGLang, or custom). Recreates delegate on backend/URL change.
OpenAICompatibleProvider
Base class for OpenAI-compatible inference servers (vLLM, SGLang)
VLLMProvider
Thin subclass for vLLM — chat, streaming, embeddings, function calling, structured output
SGLangProvider
Thin subclass for SGLang — same capabilities as vLLM, benefits from RadixAttention prefix caching
RemoteInferenceProvider
For user-managed OpenAI-compatible servers — custom URL + optional auth header via extra_headers
Validated discovery: each provider’s live model list plus a key_status verdict from a real API call — ok, not_configured, invalid_key (credential rejected), or error (unreachable). available means the provider actually answered, not merely that a key is present. ?refresh=true bypasses the 5-minute cache.
GET
/v1/models/tiers
Tier-system health: each tier’s preference list with per-candidate verdicts (ok, provider_unavailable, unknown_model, unregistered, no_quota) and the model the tier currently resolves to (null when nothing on the list is usable).
GET
/v1/models/ollama/*
Ollama model management
DELETE
/v1/models/ollama/{name}
Delete a pulled model — refused with 409 while any pod, agent, or config knob still points at it (the response lists them). Fail-closed: if the orchestrator can’t confirm zero references, the delete is rejected.
The orchestrator cross-checks every configured model reference (pod agent
pins, pod default models, task-agent models, llm.default_chat_model,
llm.cloud_fallback_model) against this validated catalog at
GET /api/v1/models/assignments; tier:* pins are verified against
/v1/models/tiers — a tier that can’t resolve on any level is flagged as a
problem instead of rubber-stamped. The dashboard’s Models page shows a
warning banner for assignments that point at retired models or dead
providers. Writes are guarded too: pinning a pod or agent to a model that no
provider serves is rejected with 422, and deleting a still-referenced local
model opens a repoint dialog in the dashboard.
LiteLLM abstraction — all provider calls go through LiteLLM for unified request/response translation
Provider auto-detection — providers are registered at startup based on available credentials (env vars, credential files, keychain)
Rate limiting — per-provider daily quotas tracked in Redis; returns HTTP 429 when exhausted
Response cache — temperature=0 requests are cached to avoid redundant API calls; cache is keyed on the full request body (excluding metadata)
Translation layer — openai_compat.py converts between OpenAI wire format and Nova’s internal CompleteRequest/CompleteResponse types
Local inference abstraction — LocalInferenceProvider wraps the active backend, reading nova:config:inference.* from Redis. Supports ollama, vllm, sglang, and custom backend types. The is_local property on ModelProvider enables inflight request counting without string matching.
Model discovery — gateway discovers models from the active backend’s /v1/models endpoint (vLLM/SGLang) or Ollama’s model list. LocalInferenceProvider maintains a dynamic set of known local models for routing decisions.
Inference metrics — the /v1/inference/stats endpoint tracks tokens per second, average latency, and request counts for the active local backend, displayed in the dashboard’s Models page.
Extra headers — OpenAICompatibleProvider supports extra_headers for custom authentication, used by RemoteInferenceProvider to pass user-configured auth to custom endpoints.