💥 OpenAI Proxy Server

📄️ Quick Start

Quick start CLI, Config, Docker

Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.

See supported Embedding Providers & Models here

Load balance multiple instances of the same model

Track Spend and create virtual keys for the proxy

Add new models + Get model info without restarting proxy.

If a call fails after num_retries, fall back to another model group.

Use this to health check all LLMs defined in your config.yaml

Modify data just before making litellm completion calls call on proxy

Cache LLM Responses

Log Proxy Input, Output, Exceptions using Custom Callbacks, Langfuse, OpenTelemetry, LangFuse, DynamoDB

Cli arguments, --host, --port, --num_workers

Dockerfile