Modal logo
FeaturedGetting started Hello, worldSimple web scraperServing web endpointsLarge language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLMCut Ministral 3 cold start times by 10x with snapshotsMaximize tokens per second in batch processing with vLLMServe an ultra-low-latency chatbot with SGLangEfficient LLM Finetuning with UnslothRun a multimodal RAG chatbot to answer questions about PDFsFine-tune an LLM to replace your CEODeploy a stateless MCP with FastMCPImages, video, & 3D Edit images with Flux KontextFine-tune Wan2.1 video models on your faceRun Flux fast with torch.compileFine-tune Flux with LoRAAnimate images with LTX-VideoGenerate video clips with LTX-VideoRun Stable Diffusion with a CLI, API, and web UIAudio Deploy a Moshi voice chatbotStream transcripts at the speed of speech using Kyutai STTMake music with ACE-StepGenerate speech with ChatterboxRun high throughput batched transcription with WhisperFine-tune Whisper to recognize new wordsReal-time communication (WebRTC) Serverless WebRTCWebRTC quickstart with FastRTCComputational biology Fold proteins with Chai-1Build a protein-folding dashboardFold proteins with Boltz-2Modal Sandboxes Run a background coding agent with OpenCodeBuild a scalable AI coding platformRun a LangGraph agent's code in a secure GPU sandboxControl a sandboxed computer with an LLMBuild a stateful, sandboxed code interpreterRun Node.js, Ruby, and more in a SandboxRun a sandboxed Jupyter notebookReinforcement Learning Train a model to solve math problems using GRPO and verlTrain a model to solve coding problems using GRPO and TRLEmbeddings Embed millions of documents with TEITurn satellite images into vectors and store them in MongoDBParallel processing and job scheduling Deploy a Hacker News SlackbotRun a Document OCR job queueServe a Document OCR web appTraining models from scratch Train an SLM with early-stopping grid search over hyperparametersRun long, resumable training jobsHosting popular libraries YOLO: Fine-tune and serve computer vision modelsBlender: Build a 3D render farmStreamlit: Run and deploy Streamlit appsComfyUI: Run Flux on ComfyUI as an APISQLite: Publish explorable data with DatasetteAlgolia: Build docsearch with a crawlerConnecting to other APIs Discord: Deploy and run a Discord BotGoogle Sheets: Sync databases and APIs to a Google SheetOpenAI: Run a RAG Q&A chatbotTailscale: Add Modal Apps to your VPNPrometheus: Publish custom metrics with PushgatewayManaging data Mount S3 buckets in Modal appsBuild your own data warehouse with DuckDB, DBT, and ModalCreate a LoRA Playground with Modal, Gradio, and S3Miscellaneous
View on GitHub

Run Facebook’s Segment Anything Model 2 (SAM 2) on Modal

This example demonstrates how to deploy Facebook’s SAM 2 on Modal. SAM2 is a powerful, flexible image and video segmentation model that can be used for various computer vision tasks like object detection, instance segmentation, and even as a foundation for more complex computer vision applications. SAM2 extends the capabilities of the original SAM to include video segmentation.

In particular, this example segments this video of a man jumping off the cliff.

The output should look something like this:

Set up dependencies for SAM 2 

First, we set up the necessary dependencies, including torch, opencv, huggingface_hub, torchvision, and the sam2 library.

We also install ffmpeg, which we will use to manipulate videos, and a Python wrapper called ffmpeg-python for a clean interface.

Wrapping the SAM 2 model in a Modal class 

Next, we define the Model class that will handle SAM 2 operations for both image and video.

We use the @modal.enter() decorators here for optimization: it makes sure the initialization method runs only once, when a new container starts, instead of in the path of every call. We’ll also use a modal Volume to cache the model weights so that they don’t need to be downloaded repeatedly when we start new containers. For more on storing model weights on Modal, see this guide.

Segmenting videos from the command line 

Finally, we define a local_entrypoint to run the segmentation from our local machine’s terminal.

There are several ways to pass files between the local machine and the Modal Function.

One way is to upload the files onto a Modal Volume, which acts as a distributed filesystem.

The other way is to convert the file to bytes and pass the bytes back and forth as the input or output of Python functions. We use this method to get the video file with the segmentation results in it back to the local machine.

Helper functions for SAM 2 inference 

Above, we used some helper functions to for some of the details, like breaking the video into frames. These are defined below.