LiteLLM统一调用多家模型

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails…

🚅 LiteLLM LiteLLM AI Gateway Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format. LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website What is LiteLLM LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format. Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization. Jump to LiteLLM Proxy (LLM Gateway) Docs Jump to Supported LLM Providers Why LiteLLM Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction: Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling Drop-in OpenAI compatibility — swap providers without rewriting your code Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box 8ms P95 latency at 1k RPS (benchmarks) OSS Adopters Netflix Features LLMs - Call 100+ LLMs (Python SDK + AI Gateway) All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more. Python SDK uv add litellm An independent litellm-core distribution provides the Python SDK with the same import litellm API and runtime dependencies. It has no optional extras, CLI entry points, or bundled dashboard. Install one SDK distribution per environment because litellm and litellm-core own overlapping Python files. Use litellm for the proxy, CLI, and optional extras Build and install core from this checkout while its release integration is pending: python scripts/build_core_distribution.py --out-dir dist/core python -m pip install dist/core/litellm_core-*.whl The builder requires Git, uv and the Rust build toolchain. It reads the release version from the root pyproject.toml, leaves source files unchanged, and produces a wheel and a self-contained sdist in dist/core. Run installation in a fresh environment without litellm from litellm import completion import os os.environ["OPENAI_API_KEY"] = "your-openai-key" os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key" # OpenAI response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}]) # Anthropic response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}]) AI Gateway (Proxy Server) Getting Started - E2E Tutorial - Setup virtual keys, make your first request uv tool install 'litellm[proxy]' litellm --model gpt-4o import openai client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000") response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) Docs: LLM Providers Agents - Invoke A2A Agents (Python SDK + AI Gateway) Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI Python SDK - A2A Protocol from litellm.a2a_protocol import A2AClient from a2a.types import SendMessageRequest, MessageSendParams from uuid import uuid4 client = A2AClient(base_url="http://localhost:10001") request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams( message={ "role": "user", "parts": [{"kind": "text", "text": "Hello!"}], "messageId": uuid4().hex, } ) ) response = await client.send_message(request) AI Gateway (Proxy Server) Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0) import httpx from a2a.client import A2ACardResolver, ClientConfig, ClientFactory from a2a.types import Message, Part, Role, SendMessageRequest from a2a.utils.constants import TransportProtocol from uuid import uuid4 base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name headers = {"Authorization": "Bearer <your-master-key>"} # LiteLLM master key or a virtual key async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client: resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url) agent_card = await resolver.get_agent_card() config = ClientConfig( httpx_client=http_client, streaming=False, supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON], ) client = ClientFactory(config).create(agent_card) request = SendMessageRequest( message=Message( message_id=uuid4().hex, role=Role.ROLE_USER, parts=[Part(text="Hello!")], ) ) async for event in client.send_message(request): populated = event.ListFields() if populated and populated[0][0].name in ("message", "msg"): print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts)) Docs: A2A Agent Gateway MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway) Python SDK - MCP Bridge from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from litellm import experimental_mcp_client import litellm server_params = StdioServerParameters(command="python", args=["mcp_server.py"]) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # Load MCP tools in OpenAI format tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai") # Use with any LiteLLM model response = await litellm.acompletion( model="gpt-4o", messages=[{"role": "user", "content": "What's 3 + 5?"}], tools=tools ) AI Gateway - MCP Gateway Step 1. Add your MCP Server to the AI Gateway Step 2. Call MCP tools via /chat/completions curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ -H 'Authorization: Bearer <your-master-key>' \ -H 'Content-Type: application/json' \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Summarize the latest open PR"}], "tools": [{ "type": "mcp", "server_url": "litellm_proxy/mcp/