Modal logo
Featured Getting started Hello, world Simple web scraper Serving Web Functions Large language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLM Cut Ministral 3 cold start times by 10x with snapshots Maximize tokens per second in batch processing with vLLM Serve an ultra-low-latency chatbot with SGLang Deploy Nemotron 3 Serve Inkling-Small with SGLang Serve DeepSeek-V4-Flash Run a multimodal RAG chatbot to answer questions about PDFs Deploy a stateless MCP with FastMCP Images, video, & 3D Edit images with Flux Kontext Fine-tune Wan2.1 video models on your face Run Flux fast with torch.compile Animate images with LTX-Video Generate video clips with LTX-Video Run Stable Diffusion with a CLI, API, and web UI Audio Stream transcripts at the speed of speech using Kyutai STT Make music with ACE-Step Generate speech with Chatterbox Run high throughput batched transcription with Whisper Fine-tune Whisper to recognize new words Real-time communication (WebRTC) Serverless WebRTC WebRTC quickstart with FastRTC Computational biology Design protein binders with ESMFold2 Fold proteins with Chai-1 Build a protein-folding dashboard Fold proteins with Boltz-2 Modal Sandboxes Run Cursor Cloud Agents on Modal Build a scalable AI coding platform Create GIFs from Slack using the Claude Agent SDK Run a LangGraph agent's code in a secure GPU sandbox Watch a computer-use agent work in real-time over VNC Run massively parallel evals on Modal Sandboxes Build a stateful, sandboxed code interpreter Run Node.js, Ruby, and more in a Sandbox Speed up Sandbox starts with warm pools Inject secrets into outbound requests Reinforcement Learning Paint flowers with code Train coding agents Make LLMs better at math Embeddings Serve Liquid AI embeddings with Modal Servers Embed millions of documents with TEI Parallel processing and job scheduling Deploy a Hacker News Slackbot Run a Document OCR job queue Serve a Document OCR web app Training models from scratch Train an SLM with early-stopping grid search over hyperparameters Run long, resumable training jobs Hosting popular libraries YOLO: Fine-tune and serve computer vision models Blender: Build a 3D render farm Streamlit: Run and deploy Streamlit apps SQLite: Publish explorable data with Datasette Algolia: Build docsearch with a crawler Connecting to other APIs Google Sheets: Sync databases and APIs to a Google Sheet OpenAI: Run a RAG Q&A chatbot Tailscale: Add Modal Apps to your VPN Prometheus: Publish custom metrics with Pushgateway Managing data Mount S3 buckets in Modal Apps Build your own data warehouse with DuckDB, dbt, and Modal Miscellaneous
View on GitHub

Deploy LangChain and LangGraph applications with LangServe

This code demonstrates how to deploy a LangServe application on Modal. LangServe makes it easy to wrap LangChain and LangGraph applications in a FastAPI server, and Modal makes it easy to deploy FastAPI servers.

The LangGraph application that it serves is from our sandboxed LLM coding agent example.

You can find the code for the agent and several other code files associated with this example in the codelangchain directory of our examples repo.

import modal

from .agent import construct_graph, create_sandbox
from .src.common import image

app = modal.App("example-codelangchain-langserve")

image = image.uv_pip_install("langserve[all]==0.3.0")


@app.function(
    image=image,
    secrets=[  # see the agent.py file for more information on Secrets
        modal.Secret.from_name("openai-secret", required_keys=["OPENAI_API_KEY"]),
        modal.Secret.from_name("langsmith-secret", required_keys=["LANGCHAIN_API_KEY"]),
    ],
)
@modal.asgi_app()
def serve():
    from fastapi import FastAPI, responses
    from fastapi.middleware.cors import CORSMiddleware
    from langchain_core.runnables import RunnableLambda
    from langserve import add_routes

    # create a FastAPI app
    web_app = FastAPI(
        title="CodeLangChain Server",
        version="1.0",
        description="Writes code and checks if it runs.",
    )

    # set all CORS enabled origins
    web_app.add_middleware(
        CORSMiddleware,
        allow_origins=["*"],
        allow_credentials=True,
        allow_methods=["*"],
        allow_headers=["*"],
        expose_headers=["*"],
    )

    def inp(question: str) -> dict:
        return {"keys": {"question": question, "iterations": 0}}

    def out(state: dict) -> str:
        if "finish" in state:
            return state["finish"]["keys"]["response"]
        elif len(state) > 0 and "finish" in state[-1]:
            return state[-1]["finish"]["keys"]["response"]
        else:
            return str(state)

    graph = construct_graph(create_sandbox(app), debug=False).compile()

    chain = RunnableLambda(inp) | graph | RunnableLambda(out)

    add_routes(
        web_app,
        chain,
        path="/codelangchain",
    )

    # redirect the root to the interactive playground
    @web_app.get("/")
    def redirect():
        return responses.RedirectResponse(url="/codelangchain/playground")

    # return the FastAPI app and Modal will deploy it for us
    return web_app