LiteLLM Proxy

LiteLLM is a high-performance open-source AI gateway that standardizes 100+ LLM providers (Anthropic, OpenAI, Bedrock, Vertex AI, Ollama, DeepSeek, Azure) into a unified OpenAI-compatible API format with automated load balancing, fallback routing, rate limiting, and centralized team spend tracking.

Website: https://docs.litellm.ai/

---

Setting Up LiteLLM Proxy

1. Start the Proxy: Run LiteLLM locally or deploy it as a Docker container:

   pip install litellm
   litellm --config /path/to/config.yaml --port 4000

2. Generate an API Key: If using the LiteLLM Proxy Admin UI (http://localhost:4000/ui), navigate to API Keys and generate a new key (sk-...). For simple local development, you can use your configured master key.

---

Supported Models & Routing

LiteLLM allows mapping custom alias names in your config.yaml to any underlying cloud or local provider:

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: deep-reasoner
    litellm_params:
      model: deepseek/deepseek-r1
      api_key: os.environ/DEEPSEEK_API_KEY
  - model_name: gpt-coding
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

Common model IDs you can specify in PostQode when routed through LiteLLM:

• claude-sonnet or anthropic/claude-sonnet-5
• gpt-5.5 or openai/gpt-5.5
• deepseek-r1 or deepseek/deepseek-r1
• Custom team aliases configured in your model_list

---

Configuration in PostQode

1. Open PostQode Settings ().
2. Select LiteLLM in the API Provider dropdown.
3. Enter your LiteLLM Proxy Base URL (e.g. http://localhost:4000/v1 or https://litellm.internal.corp/v1).
4. Enter your LiteLLM API Key (sk-...).
5. Enter the target Model Name configured in your LiteLLM config.yaml (e.g. claude-sonnet or gpt-coding).
6. Click Save Settings to connect.

---

Key Enterprise Features

• Automated Fallbacks: Automatically failover to a backup model or secondary provider region if primary provider rate limits are hit.
• Budget Tracking: Set per-developer or per-team spending caps directly inside LiteLLM.
• Prompt Caching: Passthrough prompt caching support for Anthropic and OpenAI models to minimize token costs.