Modal logo
FeaturedGetting started Hello, worldSimple web scraperServing web endpointsLarge language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLMCut Ministral 3 cold start times by 10x with snapshotsMaximize tokens per second in batch processing with vLLMServe an ultra-low-latency chatbot with SGLangEfficient LLM Finetuning with UnslothRun a multimodal RAG chatbot to answer questions about PDFsFine-tune an LLM to replace your CEODeploy a stateless MCP with FastMCPImages, video, & 3D Edit images with Flux KontextFine-tune Wan2.1 video models on your faceRun Flux fast with torch.compileFine-tune Flux with LoRAAnimate images with LTX-VideoGenerate video clips with LTX-VideoRun Stable Diffusion with a CLI, API, and web UIAudio Deploy a Moshi voice chatbotStream transcripts at the speed of speech using Kyutai STTMake music with ACE-StepGenerate speech with ChatterboxRun high throughput batched transcription with WhisperFine-tune Whisper to recognize new wordsReal-time communication (WebRTC) Serverless WebRTCWebRTC quickstart with FastRTCComputational biology Fold proteins with Chai-1Build a protein-folding dashboardFold proteins with Boltz-2Modal Sandboxes Run a background coding agent with OpenCodeBuild a scalable AI coding platformCreate GIFs from Slack using the Claude Agent SDKRun a LangGraph agent's code in a secure GPU sandboxControl a sandboxed computer with an LLMBuild a stateful, sandboxed code interpreterRun Node.js, Ruby, and more in a SandboxSpeed up Sandbox starts with warm poolsReinforcement Learning Train a model to solve math problems using GRPO and verlTrain a model to solve coding problems using GRPO and TRLEmbeddings Embed millions of documents with TEITurn satellite images into vectors and store them in MongoDBParallel processing and job scheduling Deploy a Hacker News SlackbotRun a Document OCR job queueServe a Document OCR web appTraining models from scratch Train an SLM with early-stopping grid search over hyperparametersRun long, resumable training jobsHosting popular libraries YOLO: Fine-tune and serve computer vision modelsBlender: Build a 3D render farmStreamlit: Run and deploy Streamlit appsSQLite: Publish explorable data with DatasetteAlgolia: Build docsearch with a crawlerConnecting to other APIs Discord: Deploy and run a Discord BotGoogle Sheets: Sync databases and APIs to a Google SheetOpenAI: Run a RAG Q&A chatbotTailscale: Add Modal Apps to your VPNPrometheus: Publish custom metrics with PushgatewayManaging data Mount S3 buckets in Modal appsBuild your own data warehouse with DuckDB, DBT, and ModalCreate a LoRA Playground with Modal, Gradio, and S3Miscellaneous
View on GitHub

Streaming Speaker Diarization with Sortformer2.1

In this example, we show how to deploy a streaming speaker diarization service with NVIDIA’s Sortformer2.1 on Modal. Sortformer2.1 is a state-of-the-art speaker diarization model that is designed to operate on streams of audio.

Try it yourself! Click the “View on GitHub” button to see the code. And sign up for a Modal account if you haven’t already.

Setup 

We start by importing some basic packages and the Modal SDK. As well as setting up our Modal App, Volume, and Image.

Run Sortformer2.1 speaker diarization 

Now we’re ready to add the code that runs the Sortformer2.1 speaker diarization model.

We use a Modal Cls so that we can separate out the model loading and setup code from the inference. For more on lifecycle management with Clses and cold start penalty reduction on Modal, see this guide. In particular, the Sortformer2.1 model is amenable to GPU snapshots which can significantly reduce cold start times.

We also include two configurations. The low latency configuration is used for real-time diarization, and the high latency configuration is used for non-real-time diarization with higher accuracy.

Using WebSockets to stream audio and diarization results 

We use a Modal ASGI app to serve the diarization results over WebSockets. This allows us to stream the diarization results to the client in real-time.

We use a simple queue-based architecture to handle the audio and diarization results.

The audio is received from the client over WebSockets and added to a queue. The diarization results are then processed and added to a queue. The diarization results are then sent to the client over WebSockets.

Serving the diarization results to a frontend 

We use a simple HTML frontend to display the diarization results.