I wrote about LangChain and then about Pydantic AI. Both posts ended in roughly the same place: pick the one whose shape matches your application. LiteLLM keeps coming up in the same conversations, and the framing is usually wrong, because people line it up as a third option in the same bracket.

It belongs in a different bracket. LangChain and Pydantic AI both answer the question “how should my application be structured around a model.” LiteLLM answers a much narrower question: “how do I call any model at all, from anywhere, and know what it cost.” That narrowness is the whole reason it works, and it is also the reason a real project frequently ends up with LiteLLM underneath one of the other two.

Every example here runs locally with Ollama, so there are no API keys and no per-token billing. Versions used throughout: litellm 1.98.0, pydantic 2.13.4, pydantic-ai 2.33.0, langchain 1.3.16, and langchain-ollama 1.1.0, running against llama3.2:3b.

What LiteLLM Actually Does

LiteLLM took one decision early and built everything on top of it. The OpenAI Chat Completions request and response format won, so make every provider speak it.

That is the entire product. You call litellm.completion() with an OpenAI-shaped request and a model string that names the provider, and you get an OpenAI-shaped response back. Anthropic, Bedrock, Vertex, Azure, Cohere, Together, Groq, and Ollama all arrive at the same object. Anything that already knows how to read an OpenAI response reads all of them.

Everything else in the library follows from that single normalization. If every provider speaks one dialect, you can price them against one table, retry across them, fail over between them, and put them behind one HTTP endpoint. The scale is not small:

import litellm
print("providers:", len(litellm.provider_list))
print("models priced:", len(litellm.model_cost))

providers: 149
models priced: 3175

What LiteLLM does not ship is the more useful list. There are no agents, no chains, no prompt templates, no output parsers, no document loaders, no text splitters, no vector stores, and no retrievers. It has an embedding() function, and that is where the retrieval story stops. If you want a program structure, you bring your own or you bring LangChain.

Setting Up

Install Ollama from ollama.com and pull the model:

ollama pull llama3.2:3b

Then create a virtual environment and install the library:

python3 -m venv venv
source venv/bin/activate
pip install litellm

One thing to know before you run that. pip install litellm is not a small install, and pip install "litellm[proxy]" is a great deal larger, because the proxy extra pulls in a web framework, a database client, and an auth stack. Measured in a clean virtual environment on Python 3.13:

Install	Packages	Site-packages
langchain langchain-ollama	39	58 MB
pydantic-ai	98	187 MB
litellm	56	236 MB
litellm[proxy]	108	578 MB

The reason is that litellm vendors provider SDKs and a large static price table rather than splitting integrations into optional packages the way LangChain does. That is a deliberate trade: one install gets you every provider, and you pay for it in container size. If you build slim images, size it before you commit.

Your First Call

One function, and a model string that carries the provider:

import litellm

resp = litellm.completion(
    model="ollama_chat/llama3.2:3b",
    messages=[
        {"role": "system", "content": "You are a terse network engineer. Answer in one sentence."},
        {"role": "user", "content": "What is the difference between a VLAN and a subnet?"},
    ],
)
print(resp.choices[0].message.content)

A VLAN is a virtual network that shares the same physical network infrastructure but is logically separated, whereas a subnet is a sub-network that shares the same physical network infrastructure but has its own unique IP address range.

Notice what is missing compared to the other two frameworks. There is no client object, no model object, and no agent. There is a module-level function and a string. The string is the provider routing: ollama_chat/llama3.2:3b, anthropic/claude-opus-4-5, bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0. Changing providers means changing that string and having the right credential in the environment.

The messages are plain dictionaries with role and content. There is no SystemMessage class and no instructions parameter, because the OpenAI wire format is the abstraction. That is worth pausing on, since it is the single biggest ergonomic difference from the other two libraries. LangChain gives you message classes. Pydantic AI gives you an agent that owns its instructions. LiteLLM gives you the raw payload, and you get exactly what you build.

The response is a Pydantic model shaped like an OpenAI response:

print(repr(resp))

ModelResponse(id='chatcmpl-bc5483e2-0ecf-4fbc-848c-53e327c18292', created=1787445441, model='ollama_chat/llama3.2:3b', object='chat.completion', choices=[Choices(finish_reason='stop', index=0, message=Message(content='A VLAN is a virtual network...', role='assistant', tool_calls=None, function_call=None))], usage=Usage(completion_tokens=42, prompt_tokens=48, total_tokens=90))

The usage block is normalized the same way. Ollama reports prompt_eval_count, Anthropic reports input_tokens, and both land on usage.prompt_tokens.

There is a detail in the provider prefix worth knowing. LiteLLM has both ollama/ and ollama_chat/. The first uses Ollama’s older /api/generate endpoint, and the second uses /api/chat. Use ollama_chat/ for anything conversational or tool-using. Use ollama/ for embeddings, which is what the embedding examples below do.

Memory Is Still Yours

Nothing in LiteLLM holds a session. Each call is independent, and the message list is the memory, exactly as in the other two frameworks. The difference is that you append dictionaries instead of objects:

first = litellm.completion(model="ollama_chat/llama3.2:3b", messages=messages, temperature=0.0)
print(first.choices[0].message.content)

messages.append({"role": "assistant", "content": first.choices[0].message.content})
messages.append({"role": "user", "content": "Which one does a router care about?"})

second = litellm.completion(model="ollama_chat/llama3.2:3b", messages=messages, temperature=0.0)
print(second.choices[0].message.content)
print("input tokens:", first.usage.prompt_tokens, "->", second.usage.prompt_tokens)

A VLAN is a logical grouping of devices on a single network, while a subnet is a physical division of an IP network into smaller, isolated segments.
A router cares about subnets, as it uses subnetting to route traffic between different networks.
input tokens: 48 -> 96

The same 48 tokens went out on the first call in all three frameworks, and the same growth shows up on the second. Every turn resends the whole history. No library removes that cost, and any library that claims to has quietly started summarizing or truncating on your behalf.

Structured Output Works, and Stops Halfway

Pass a Pydantic model as response_format and LiteLLM converts it to a JSON Schema and sends it to the provider:

import litellm
from pydantic import BaseModel, Field


class Ticket(BaseModel):
    device: str = Field(description="The hostname of the affected device")
    severity: int = Field(description="Severity from 1 (critical) to 5 (cosmetic)", ge=1, le=5)
    component: str = Field(description="The failing component, such as interface, bgp, power")
    summary: str = Field(description="A one line summary of the problem")


TEXT = (
    "Overnight the core switch core-sw-01 dropped BGP session to its upstream "
    "peer three times. Each flap lasted about 40 seconds. Customer traffic was "
    "affected. Please look at this today."
)

resp = litellm.completion(
    model="ollama_chat/llama3.2:3b",
    messages=[{"role": "user", "content": TEXT}],
    response_format=Ticket,
    temperature=0.0,
)
raw = resp.choices[0].message.content
print(type(raw))
print(raw)

<class 'str'>
{ "device": "core-sw-01", "severity": 3, "component": "BGP", "summary": "BGP session flaps with upstream peer" }

The model produced correct JSON with no prompt engineering, because the schema reached Ollama’s grammar-constrained decoder. What came back to Python is a string. LiteLLM handed the schema down and handed the text back up, and the round trip stops there:

ticket = Ticket.model_validate_json(raw)
print(repr(ticket))

Ticket(device='core-sw-01', severity=3, component='BGP', summary='BGP session flaps with upstream peer')

One extra line, which is a fair price. The part that costs more is what a type checker sees. In the Pydantic AI post I ran reveal_type across all three. Here is the third data point:

content = resp.choices[0].message.content
reveal_type(content)

lt_typed.py:14: note: Revealed type is "Any"

Pydantic AI revealed Ticket. LangChain revealed dict[Any, Any] | BaseModel. LiteLLM reveals Any, and so does the whole response object, because litellm.completion() is declared to return Any so it can also return a stream, an async result, or a mocked value from the same signature.

Any is worse than a union, because a union at least makes mypy complain. Any makes it agree with everything:

resp = litellm.completion(model="ollama_chat/llama3.2:3b", messages=[{"role": "user", "content": "x"}])
print(resp.choices[0].message.contnet)
print(resp.usage.prompt_tokns)

Success: no issues found in 1 source file

Two typos, both fatal at runtime, both invisible to the type checker. The identical script written against Pydantic AI produced two attr-defined errors before the model was ever called. If your codebase runs mypy in CI, understand that the LiteLLM boundary is a hole in that coverage, and close it yourself by validating into a model as soon as the response arrives.

Tool Calling Works, and You Write the Loop

LiteLLM normalizes tool calling across providers, which is genuinely hard and genuinely valuable. Anthropic, OpenAI, Gemini, and Ollama all describe tools differently and return calls differently. You write the OpenAI form once:

import json
import litellm

INVENTORY = {
    "core-sw-01": {"site": "dfw", "role": "core", "os": "ios-xe 17.9.4"},
    "edge-rtr-02": {"site": "aus", "role": "edge", "os": "junos 22.4R3"},
}

tools = [{
    "type": "function",
    "function": {
        "name": "get_device",
        "description": "Look up a device in the inventory by hostname.",
        "parameters": {
            "type": "object",
            "properties": {
                "hostname": {"type": "string", "description": "The device hostname."}
            },
            "required": ["hostname"],
        },
    },
}]

messages = [
    {"role": "system", "content": "You answer questions about network devices. Use the tools to look up facts. Never guess."},
    {"role": "user", "content": "What OS does edge-rtr-02 run?"},
]

resp = litellm.completion(model="ollama_chat/llama3.2:3b", messages=messages, tools=tools, temperature=0.0)
msg = resp.choices[0].message
print("finish_reason:", resp.choices[0].finish_reason)
print(msg.tool_calls)

finish_reason: tool_calls
[ChatCompletionMessageToolCall(function=Function(arguments='{"hostname": "edge-rtr-02"}', name='get_device'), id='call_r2c6qq55', type='function')]

The model asked for the tool. Now you dispatch it, append the result, and call again:

messages.append(msg.model_dump())
for tc in msg.tool_calls:
    args = json.loads(tc.function.arguments)
    result = INVENTORY.get(args["hostname"], {"error": "not found"})
    print(f"  [tool] {tc.function.name}({args})")
    messages.append({
        "role": "tool",
        "tool_call_id": tc.id,
        "name": tc.function.name,
        "content": json.dumps(result),
    })

final = litellm.completion(model="ollama_chat/llama3.2:3b", messages=messages, tools=tools, temperature=0.0)
print(final.choices[0].message.content)

  [tool] get_device({'hostname': 'edge-rtr-02'})
The edge-rtr-02 device runs Junos 22.4R3 operating system.

Identical answer to both other frameworks, from the same 3B model. The comparison is in the surrounding code. Pydantic AI needed a decorated function and a docstring, and ran the loop for you. LangChain needed @tool and create_agent(), and ran the loop for you. LiteLLM needs a hand-written JSON Schema, a hand-written dispatch table, and a hand-written loop that keeps going while finish_reason stays tool_calls.

The code above is the happy path with a single call and a single round. A production version handles parallel tool calls, tools that raise, a model that asks for a tool that does not exist, and a turn limit so a confused model cannot loop forever. That is the agent loop, and writing it is a real afternoon.

Validation Retry Is Also Yours

The Pydantic AI post used ModelRetry to reject a hostname that parsed correctly but did not exist in the inventory. The framework fed the error back to the model, which corrected itself. Here is the same task and the same outcome in LiteLLM:

import litellm
from pydantic import BaseModel, ValidationError

KNOWN = ["core-sw-01", "edge-rtr-02"]


class Change(BaseModel):
    device: str
    action: str


messages = [
    {"role": "system", "content": "Extract the requested change. Use the exact inventory hostname."},
    {"role": "user", "content": "Shut the uplink on the Dallas core switch, coresw1."},
]

result = None
for attempt in range(3):
    resp = litellm.completion(
        model="ollama_chat/llama3.2:3b",
        messages=messages,
        response_format=Change,
        temperature=0.0,
    )
    raw = resp.choices[0].message.content
    messages.append({"role": "assistant", "content": raw})
    try:
        candidate = Change.model_validate_json(raw)
    except ValidationError as exc:
        messages.append({"role": "user", "content": f"That was not valid: {exc}"})
        continue
    print(f"  attempt -> device={candidate.device!r}")
    if candidate.device not in KNOWN:
        messages.append({
            "role": "user",
            "content": f"{candidate.device!r} is not a valid hostname. "
                       f"You must copy one of these exactly: {KNOWN}",
        })
        continue
    result = candidate
    break

if result is None:
    raise RuntimeError("model never produced a valid device")
print(result)
print("model calls:", attempt + 1)

  attempt -> device='coresw1'
  attempt -> device='core-sw-01'
device='core-sw-01' action='shutdown'
model calls: 2

Same behavior, same correction, same number of model calls. Twenty-five lines of loop replace one decorated validator function and a retries=3 argument. There is nothing wrong with those twenty-five lines, and I would rather read them than debug a framework’s retry semantics at two in the morning. Write them once per project, though, not once per extraction.

Where LiteLLM Excels

Everything above makes LiteLLM look like the low-level option, which it is. The features below are the ones neither of the other two libraries has, and they are the reason to reach for it.

One exception hierarchy across 149 providers

Every provider fails differently, and every SDK raises its own types. LiteLLM maps all of them onto the OpenAI SDK’s exception classes, so the errors are not merely similar, they are the same classes:

try:
    litellm.completion(model="gpt-5", messages=[{"role": "user", "content": "hi"}], api_key="sk-bad")
except Exception as e:
    import openai
    print(type(e).__name__, "| openai.AuthenticationError:", isinstance(e, openai.AuthenticationError))

AuthenticationError | openai.AuthenticationError: True

One except RateLimitError block covers Anthropic throttling, Bedrock throttling, and Vertex quota exhaustion. If you have ever written provider-specific error handling, you know what that is worth.

It knows what each provider supports

LiteLLM carries a capability table, so you can ask before you send:

from litellm import get_supported_openai_params, supports_function_calling, get_max_tokens

print(get_supported_openai_params(model="llama3.2:3b", custom_llm_provider="ollama_chat"))
print("gpt-5 max output:", get_max_tokens("gpt-5"), "| tools:", supports_function_calling(model="gpt-5"))

['max_tokens', 'max_completion_tokens', 'stream', 'top_p', 'temperature', 'seed', 'frequency_penalty', 'stop', 'tools', 'tool_choice', 'functions', 'response_format', 'reasoning_effort']
gpt-5 max output: 128000 | tools: True

Send a parameter the provider cannot take and you get a clear failure rather than a silent drop or a provider-side 400:

litellm.completion(model="ollama_chat/llama3.2:3b", messages=msgs, presence_penalty=0.5)

litellm.UnsupportedParamsError: ollama_chat does not support parameters: ['presence_penalty'],
for model=llama3.2:3b. To drop these, set `litellm.drop_params=True`

Set litellm.drop_params = True and the same code runs everywhere, with unsupported parameters silently removed per provider. That flag is what makes “swap the model string” a true statement rather than an aspiration, because otherwise every provider needs a different parameter set.

Cost accounting is built in

This is the feature I have never seen done as well anywhere else. LiteLLM ships a price table for 3175 models and attaches a cost to every response:

from litellm import completion_cost, cost_per_token

resp = litellm.completion(model="ollama_chat/llama3.2:3b", messages=msgs, temperature=0.0)
print("this call cost:", completion_cost(completion_response=resp))

for m in ["gpt-5", "claude-opus-4-5", "gemini/gemini-2.5-pro"]:
    pin, pout = cost_per_token(model=m, prompt_tokens=resp.usage.prompt_tokens,
                               completion_tokens=resp.usage.completion_tokens)
    print(f"{m}: ${pin + pout:.8f}")

this call cost: 0.0
gpt-5: $0.00006500
claude-opus-4-5: $0.00021500
gemini/gemini-2.5-pro: $0.00006500

The local call is free, which is correct. The interesting part is the second block: you can price a workload against providers you are not currently using, from the token counts you already have. Run your evaluation set against Ollama, then ask what the same volume would cost on three hosted models before you migrate. Neither LangChain nor Pydantic AI answers that question without a spreadsheet.

The Router does failover, retries, and load balancing

Router wraps a list of deployments and moves traffic between them. Here the primary has a deliberately invalid key, and the local model catches the request:

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "chat",
         "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-deliberately-invalid"}},
        {"model_name": "chat-local",
         "litellm_params": {"model": "ollama_chat/llama3.2:3b", "temperature": 0.0}},
    ],
    fallbacks=[{"chat": ["chat-local"]}],
    num_retries=0,
)

resp = router.completion(model="chat", messages=[
    {"role": "user", "content": "Name the seven OSI layers, bottom to top. List only."}])
print("answered by:", resp.model)
print(resp.choices[0].message.content)

answered by: ollama_chat/llama3.2:3b
1. Physical
2. Data Link
3. Network
4. Transport
5. Session
6. Presentation
7. Application

The caller asked for chat and never learned that the primary failed. The same object handles several deployments sharing one model_name for load balancing, per-deployment rate limits, latency-based and least-busy routing strategies, and cooldowns for deployments that are failing. Cross-provider failover is the reason many teams adopt LiteLLM at all, because it turns a provider outage into a latency bump.

Caching, mocking, and batching

Three small utilities that pay for themselves quickly. Response caching keys on the request:

import litellm, time
from litellm.caching.caching import Cache

litellm.cache = Cache(type="local")
for i in (1, 2):
    t = time.perf_counter()
    r = litellm.completion(model="ollama_chat/llama3.2:3b", messages=msgs, temperature=0.0, caching=True)
    print(f"call {i}: {time.perf_counter() - t:.3f}s  cache_hit={r._hidden_params.get('cache_hit')}")

call 1: 3.752s  cache_hit=None
call 2: 0.007s  cache_hit=True

Swap type="local" for type="redis" and the cache is shared across processes. mock_response returns a canned answer without contacting any model, which makes the LLM layer testable in CI with no network and no key:

r = litellm.completion(model="anthropic/claude-opus-4-5", messages=msgs,
                       mock_response="canned answer for the test suite")
print(r.choices[0].message.content, "|", r.model)

canned answer for the test suite | claude-opus-4-5

And batch_completion runs a list of message lists concurrently:

batch = litellm.batch_completion(
    model="ollama_chat/llama3.2:3b",
    messages=[
        [{"role": "user", "content": "Define MTU in five words."}],
        [{"role": "user", "content": "Define VLAN in five words."}],
        [{"role": "user", "content": "Define BGP in five words."}],
    ],
    temperature=0.0,
)

3 results in 2.0s
 - Maximum transmission unit size.
 - Virtual Local Area Network segment.
 - Border Gateway Protocol routing.

The proxy is the real product

The Python library is the part people meet first. The part organizations actually deploy is the proxy, which puts everything above behind an HTTP endpoint that speaks OpenAI. A minimal config:

model_list:
  - model_name: local-chat
    litellm_params:
      model: ollama_chat/llama3.2:3b
      api_base: http://localhost:11434

litellm_settings:
  drop_params: true

litellm --config config.yaml --port 4000

Now anything that speaks OpenAI can reach it, including plain curl:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"What does MTU stand for? One sentence."}],"temperature":0}'

{
  "model": "local-chat",
  "choices": [{"finish_reason": "stop", "index": 0,
    "message": {"content": "MTU stands for Maximum Transmission Unit, which refers to the largest amount of data that can be transmitted over a network connection at one time.", "role": "assistant"}}],
  "usage": {"completion_tokens": 29, "prompt_tokens": 35, "total_tokens": 64}
}

With a database attached, the proxy adds virtual API keys per team, per-key budgets and rate limits, spend reporting, audit logs, and guardrail hooks. That combination is why the proxy shows up in companies that need a single point where model access is granted, priced, and logged. A platform team runs one proxy, application teams get keys, and nobody ships a provider credential in an application container.

Where LiteLLM Lags

Collecting the gaps in one place, because they are consistent.

Nothing is typed. completion() returns Any, so every attribute access downstream is unchecked. This is the sharpest reversal against Pydantic AI, where the concrete output type flows through to .output and typos fail at edit time.

No agent loop. Tool calling is normalized, and the loop that makes tool calling useful is yours. Multi-step tool use, parallel calls, tool errors, and turn limits are all your code.

No validation retry. A schema violation is an exception you catch. Feeding the error back to the model is a loop you write.

No retrieval anything. There is embedding(), and no document loaders, no splitters, no vector stores, and no retrievers. If the application is fundamentally about searching a pile of documents, LangChain has already written the connector and LiteLLM has not.

No prompt management. No templates, no partials, no message classes. You build dictionaries and format strings.

Heavy install. 236 MB for the library, 578 MB with the proxy extra, against 58 MB for LangChain with the Ollama integration.

The price table has edges. The metadata that makes the cost tracking good only covers models in the table, and local models are not in it:

from litellm import get_max_tokens
get_max_tokens("ollama_chat/llama3.2:3b")

Exception: Model llama3.2:3b isn't mapped yet.

You can register a model yourself with litellm.register_model(), and the point stands that the helpers are only as good as the table behind them. The table is community-maintained JSON, which means it is broad and occasionally stale. Treat reported costs as very good estimates rather than as an invoice.

The Comparison

	LiteLLM	Pydantic AI	LangChain
Core abstraction	completion(), a function	Agent, generic over deps and output	Runnable, composed with |
Provider coverage	149, one install	Broad, one install	Broad, split into packages
Message format	OpenAI dicts	Framework message objects	Message classes
Structured output	Schema sent, string returned	Typed instance returned	with_structured_output(), union returned
reveal_type of the result	Any	Ticket	dict[Any, Any] | BaseModel
Tool calling	Normalized, loop is yours	Decorator, loop is framework’s	Decorator, loop is framework’s
Validation retry	Your loop	ModelRetry	Your code
Cross-provider failover	Router, first class	None	None
Cost tracking	3175 models priced, per call	Usage tokens only	Usage tokens only
Response caching	Local, Redis, S3, semantic	None built in	Yes
Rate limits and budgets	Yes, in the proxy	None	None
Prompt templates	None	Instructions on the agent	Extensive
Retrieval and vector stores	None	None	Extensive
Gateway or server	Yes, the proxy is the flagship	None	LangServe
Install footprint	236 MB, 578 MB with proxy	187 MB	58 MB with Ollama

They Compose, Which Is the Point

The bracket error at the top of this post has a practical consequence. Because the LiteLLM proxy speaks OpenAI, both other frameworks point at it with a base URL and treat it as a provider. LangChain, through langchain-openai:

from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model="local-chat",
    base_url="http://localhost:4000/v1",
    api_key="sk-1234",
    temperature=0.0,
)
print(model.invoke("What is a VLAN? One sentence.").content)

A VLAN (Virtual Local Area Network) is a logical division of a physical network into multiple, isolated segments that share the same physical infrastructure, allowing for more efficient and secure network management.

Pydantic AI, through its OpenAI provider, running the same Ticket extraction from earlier in this post:

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIChatModel(
    "local-chat",
    provider=OpenAIProvider(base_url="http://localhost:4000/v1", api_key="sk-1234"),
)
agent = Agent(model, output_type=Ticket)
print(repr(agent.run_sync(TEXT).output))

Ticket(device='core-sw-01', severity=3, component='BGP session', summary='core-sw-01 BGP session to upstream peer dropped three times, causing customer traffic impact')

Both frameworks got their usual behavior. Pydantic AI still returned a typed Ticket, because the typing happens in Pydantic AI and the routing happens in LiteLLM. Meanwhile the proxy logged the request, applied the key’s budget, and priced the call.

That is the architecture worth copying. LiteLLM owns provider access, credentials, cost, and failover. The application framework owns program structure. Neither one has to know much about the other, and the team that runs the proxy is usually not the team that writes the agent.

Best Use Cases

Reach for LiteLLM directly when the model call is the whole job. A classifier, an extractor, a summarizer, or a batch job that runs one prompt over ten thousand rows. A single function call and a dictionary is less machinery than an agent, and you should not install a framework to avoid writing four lines.

Reach for it when you need more than one provider at once. Cross-provider failover, load balancing across regions or accounts, or an application that lets the user pick a model. Router is the only one of the three libraries that treats this as a first-class problem.

Reach for it when somebody is going to ask what this costs. The price table, completion_cost(), and the proxy’s spend reporting are the strongest answer in this space, and the ability to price a workload against providers you have not adopted yet is genuinely useful before a migration.

Reach for the proxy when more than one team calls models. One endpoint, virtual keys, per-team budgets, central logging, and no provider credentials in application containers. This is an infrastructure problem, and the proxy is infrastructure.

Reach for it when you are running local and hosted models side by side. One code path covers Ollama in development and Bedrock in production, with drop_params absorbing the differences.

Reach for something else when you need program structure. If you want typed output that a type checker can see, use Pydantic AI. If you want retrieval over documents, use LangChain. If you want an agent loop you did not write, use either one. LiteLLM is deliberately not in that business, and pretending otherwise means reimplementing one of those two libraries badly.

The most useful way to test the claim is to take whatever LLM call your codebase makes most often and put the LiteLLM proxy in front of it, without changing a line of application code beyond a base URL and a key. Point your existing LangChain or Pydantic AI code at http://localhost:4000/v1 and confirm the behavior is unchanged. Then look at what the proxy logged. That takes about twenty minutes, and it tells you whether the gateway layer is missing from your stack.