Modal logo
FeaturedGetting started Hello, worldSimple web scraperServing web endpointsLarge language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLMCut Ministral 3 cold start times by 10x with snapshotsMaximize tokens per second in batch processing with vLLMServe an ultra-low-latency chatbot with SGLangEfficient LLM Finetuning with UnslothRun a multimodal RAG chatbot to answer questions about PDFsFine-tune an LLM to replace your CEODeploy a stateless MCP with FastMCPImages, video, & 3D Edit images with Flux KontextFine-tune Wan2.1 video models on your faceRun Flux fast with torch.compileFine-tune Flux with LoRAAnimate images with LTX-VideoGenerate video clips with LTX-VideoRun Stable Diffusion with a CLI, API, and web UIAudio Deploy a Moshi voice chatbotStream transcripts at the speed of speech using Kyutai STTMake music with ACE-StepGenerate speech with ChatterboxRun high throughput batched transcription with WhisperFine-tune Whisper to recognize new wordsReal-time communication (WebRTC) Serverless WebRTCWebRTC quickstart with FastRTCComputational biology Fold proteins with Chai-1Build a protein-folding dashboardFold proteins with Boltz-2Modal Sandboxes Run a background coding agent with OpenCodeBuild a scalable AI coding platformRun a LangGraph agent's code in a secure GPU sandboxControl a sandboxed computer with an LLMBuild a stateful, sandboxed code interpreterRun Node.js, Ruby, and more in a SandboxRun a sandboxed Jupyter notebookReinforcement Learning Train a model to solve math problems using GRPO and verlTrain a model to solve coding problems using GRPO and TRLEmbeddings Embed millions of documents with TEITurn satellite images into vectors and store them in MongoDBParallel processing and job scheduling Deploy a Hacker News SlackbotRun a Document OCR job queueServe a Document OCR web appTraining models from scratch Train an SLM with early-stopping grid search over hyperparametersRun long, resumable training jobsHosting popular libraries YOLO: Fine-tune and serve computer vision modelsBlender: Build a 3D render farmStreamlit: Run and deploy Streamlit appsSQLite: Publish explorable data with DatasetteAlgolia: Build docsearch with a crawlerConnecting to other APIs Discord: Deploy and run a Discord BotGoogle Sheets: Sync databases and APIs to a Google SheetOpenAI: Run a RAG Q&A chatbotTailscale: Add Modal Apps to your VPNPrometheus: Publish custom metrics with PushgatewayManaging data Mount S3 buckets in Modal appsBuild your own data warehouse with DuckDB, DBT, and ModalCreate a LoRA Playground with Modal, Gradio, and S3Miscellaneous
View on GitHub

Serve the Qwen3.5 Vision-Language Model with SGLang

Vision-Language Models (VLMs) are like LLMs with eyes: they can generate text based not just on other text, but on images as well.

This example shows how to serve a VLM on Modal using the SGLang library with an OpenAI-compatible API server.

Setup and container image definition 

First, we import our global dependencies and define constants.

To define the container Image with our server’s dependencies, we build off of the official SGLang Docker image with CUDA 12.9.

Configure the model 

Qwen3.5-35B-A3B-FP8 is a vision-language reasoning foundational model with 35B total parameters, of which only 3B are activated per input sequence per forward pass. We use the 8bit quantized floating point version of the model for faster cold starts and faster inference with negligible behavior differences.

Configure GPU 

We use a single H100 GPU. The 35 GB of model weights fits comfortably in this GPU’s 80GB of high-bandwidth memory.

Cacheing in Modal Volumes 

Modal Apps typically cache some artifacts in a Modal Volume for faster cold starts. Here, we cache the model weights and the JIT-compiled DeepGEMM kernels.

We additionally compile the DeepGEMM kernels as part of building the container Image. This can take tens of minutes the first time, but only takes seconds when reading from cache.

Define the inference server 

With environment setup out of the way, we’re ready to define our inference server. We use a Modal Cls to separate container startup logic from input processing (as part of modal.enter-decorated methods). We use a Modal HTTP Server to create a low latency edge deployment in the us served by a proxy in us-east. We also handle clean teardown of the server in a modal.exit method.

Setting up the server 

The server configuration is based on the information in the Hugging Face repo. It includes speculative decoding via multi-token prediction for improved performance at low to moderate concurrency. For more on optimizing the performance of VLMs and LLMs, see this guide.

Before returning from our modal.enter method, we wait for the server to finish spinning up, which can take several minutes.

We also send a few warmup requests to ensure that the server is fully ready to service requests — otherwise the first few requests to a new replica might be substantially slower.

Test the server 

We can test the entire server creation, from soup to nuts, by running the file with modal run. We just need to add a local_entrypoint that exercises the server.

The client logic is normally handled by your preferred interface — a coding agent harness like OpenCode, a chat UI in the browser. Our server uses the standard OpenAI-compatible API format, so most of these clients should work out of the box. We replicate the minimum amount of its functionality we need for a test below.

Note that in the probe we include a Modal-Session-Id header for sticky routing between Modal HTTP Server replicas and ignore 503s that occur when no Modal HTTP Server replicas are available.

You can kick off a test run with the command

Deploy the server 

When you’re ready to deploy the server, replace modal run with modal deploy: