LangServe API for LLM Applications

What is LangServe?

LangServe is a deployment framework from LangChain that turns any LangChain chain, agent, or runnable into a production-ready REST API. It provides automatic API documentation, streaming support, and a playground for testing.

Installation

pip install langserve[all] langchain langchain-openai fastapi uvicorn

Creating a LangServe API

from fastapi import FastAPI
from langserve import add_routes
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

app = FastAPI(title="My LLM API")

# Define a chain
prompt = ChatPromptTemplate.from_template(
    "You are a helpful assistant. {question}"
)
llm = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | llm

# Add routes
add_routes(app, chain, path="/chat")

# Run: uvicorn server:app --host 0.0.0.0 --port 8000

Features

• Auto-generated API docs at /docs
• Streaming support via Server-Sent Events
• Built-in playground at /chat/playground
• Input/output type validation
• Batch endpoint for multiple inputs
• Works with any LangChain runnable

Production Deployment

# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

# Use gunicorn with uvicorn workers for production
CMD ["gunicorn", "server:app", "-w", "4", "-k", "uvicorn.workers.UvicornWorker", "--bind", "0.0.0.0:8000"]