Modal logo
FeaturedGetting started Hello, worldSimple web scraperServing web endpointsLarge language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLMCut Ministral 3 cold start times by 10x with snapshotsMaximize tokens per second in batch processing with vLLMServe an ultra-low-latency chatbot with SGLangEfficient LLM Finetuning with UnslothRun a multimodal RAG chatbot to answer questions about PDFsFine-tune an LLM to replace your CEODeploy a stateless MCP with FastMCPImages, video, & 3D Edit images with Flux KontextFine-tune Wan2.1 video models on your faceRun Flux fast with torch.compileFine-tune Flux with LoRAAnimate images with LTX-VideoGenerate video clips with LTX-VideoRun Stable Diffusion with a CLI, API, and web UIAudio Deploy a Moshi voice chatbotStream transcripts at the speed of speech using Kyutai STTMake music with ACE-StepGenerate speech with ChatterboxRun high throughput batched transcription with WhisperFine-tune Whisper to recognize new wordsReal-time communication (WebRTC) Serverless WebRTCWebRTC quickstart with FastRTCComputational biology Fold proteins with Chai-1Build a protein-folding dashboardFold proteins with Boltz-2Modal Sandboxes Run a background coding agent with OpenCodeBuild a scalable AI coding platformRun a LangGraph agent's code in a secure GPU sandboxControl a sandboxed computer with an LLMBuild a stateful, sandboxed code interpreterRun Node.js, Ruby, and more in a SandboxRun a sandboxed Jupyter notebookReinforcement Learning Train a model to solve math problems using GRPO and verlTrain a model to solve coding problems using GRPO and TRLEmbeddings Embed millions of documents with TEITurn satellite images into vectors and store them in MongoDBParallel processing and job scheduling Deploy a Hacker News SlackbotRun a Document OCR job queueServe a Document OCR web appTraining models from scratch Train an SLM with early-stopping grid search over hyperparametersRun long, resumable training jobsHosting popular libraries YOLO: Fine-tune and serve computer vision modelsBlender: Build a 3D render farmStreamlit: Run and deploy Streamlit appsComfyUI: Run Flux on ComfyUI as an APISQLite: Publish explorable data with DatasetteAlgolia: Build docsearch with a crawlerConnecting to other APIs Discord: Deploy and run a Discord BotGoogle Sheets: Sync databases and APIs to a Google SheetOpenAI: Run a RAG Q&A chatbotTailscale: Add Modal Apps to your VPNPrometheus: Publish custom metrics with PushgatewayManaging data Mount S3 buckets in Modal appsBuild your own data warehouse with DuckDB, DBT, and ModalCreate a LoRA Playground with Modal, Gradio, and S3Miscellaneous
View on GitHub

Fast Whisper inference using dynamic batching

In this example, we demonstrate how to run dynamically batched inference for OpenAI’s speech recognition model, Whisper, on Modal. Batching multiple audio samples together or batching chunks of a single audio sample can help to achieve a 2.8x increase in inference throughput on an A10G!

We will be running the Whisper Large V3 model. To run any of the other HuggingFace Whisper models, simply replace the MODEL_NAME and MODEL_REVISION variables.

Setup 

Let’s start by importing the Modal client and defining the model that we want to serve.

Define a container image 

We’ll start with Modal’s baseline debian_slim image and install the relevant libraries.

Caching the model weights 

We’ll define a function to download the model and cache it in a volume. You can modal run against this function prior to deploying the App.

The model class 

The inference function is best represented using Modal’s class syntax.

We define a @modal.enter method to load the model when the container starts, before it picks up any inputs. The weights will be loaded from the Hugging Face cache volume so that we don’t need to download them when we start a new container. For more on storing model weights on Modal, see this guide.

We also define a transcribe method that uses the @modal.batched decorator to enable dynamic batching. This allows us to invoke the function with individual audio samples, and the function will automatically batch them together before running inference. Batching is critical for making good use of the GPU, since GPUs are designed for running parallel operations at high throughput.

The max_batch_size parameter limits the maximum number of audio samples combined into a single batch. We used a max_batch_size of 64, the largest power-of-2 batch size that can be accommodated by the 24 A10G GPU memory. This number will vary depending on the model and the GPU you are using.

The wait_ms parameter sets the maximum time to wait for more inputs before running the batched transcription. To tune this parameter, you can set it to the target latency of your application minus the execution time of an inference batch. This allows the latency of any request to stay within your target latency.

Transcribe a dataset 

In this example, we use the librispeech_asr_dummy dataset from Hugging Face’s Datasets library to test the model.

We use map.aio to asynchronously map over the audio files. This allows us to invoke the batched transcription method on each audio sample in parallel.

Run the model 

We define a local_entrypoint to run the transcription. You can run this locally with modal run batched_whisper.py.