Modal logo
FeaturedGetting started Hello, worldSimple web scraperServing web endpointsLarge language models (LLMs) Deploy an OpenAI-compatible LLM service with vLLMCut Ministral 3 cold start times by 10x with snapshotsMaximize tokens per second in batch processing with vLLMServe an ultra-low-latency chatbot with SGLangEfficient LLM Finetuning with UnslothRun a multimodal RAG chatbot to answer questions about PDFsFine-tune an LLM to replace your CEODeploy a stateless MCP with FastMCPImages, video, & 3D Edit images with Flux KontextFine-tune Wan2.1 video models on your faceRun Flux fast with torch.compileFine-tune Flux with LoRAAnimate images with LTX-VideoGenerate video clips with LTX-VideoRun Stable Diffusion with a CLI, API, and web UIAudio Deploy a Moshi voice chatbotStream transcripts at the speed of speech using Kyutai STTMake music with ACE-StepGenerate speech with ChatterboxRun high throughput batched transcription with WhisperFine-tune Whisper to recognize new wordsReal-time communication (WebRTC) Serverless WebRTCWebRTC quickstart with FastRTCComputational biology Fold proteins with Chai-1Build a protein-folding dashboardFold proteins with Boltz-2Modal Sandboxes Run a background coding agent with OpenCodeBuild a scalable AI coding platformRun a LangGraph agent's code in a secure GPU sandboxControl a sandboxed computer with an LLMBuild a stateful, sandboxed code interpreterRun Node.js, Ruby, and more in a SandboxRun a sandboxed Jupyter notebookReinforcement Learning Train a model to solve math problems using GRPO and verlTrain a model to solve coding problems using GRPO and TRLEmbeddings Embed millions of documents with TEITurn satellite images into vectors and store them in MongoDBParallel processing and job scheduling Deploy a Hacker News SlackbotRun a Document OCR job queueServe a Document OCR web appTraining models from scratch Train an SLM with early-stopping grid search over hyperparametersRun long, resumable training jobsHosting popular libraries YOLO: Fine-tune and serve computer vision modelsBlender: Build a 3D render farmStreamlit: Run and deploy Streamlit appsComfyUI: Run Flux on ComfyUI as an APISQLite: Publish explorable data with DatasetteAlgolia: Build docsearch with a crawlerConnecting to other APIs Discord: Deploy and run a Discord BotGoogle Sheets: Sync databases and APIs to a Google SheetOpenAI: Run a RAG Q&A chatbotTailscale: Add Modal Apps to your VPNPrometheus: Publish custom metrics with PushgatewayManaging data Mount S3 buckets in Modal appsBuild your own data warehouse with DuckDB, DBT, and ModalCreate a LoRA Playground with Modal, Gradio, and S3Miscellaneous
View on GitHub

LoRAs Galore: Create a LoRA Playground with Modal, Gradio, and S3

This example shows how to mount an S3 bucket in a Modal app using CloudBucketMount. We will download a bunch of LoRA adapters from the HuggingFace Hub into our S3 bucket then read from that bucket, on the fly, when doing inference.

By default, we use the IKEA instructions LoRA as an example, which produces the following image when prompted to generate “IKEA instructions for building a GPU rig for deep learning”:

IKEA instructions for building a GPU rig for deep learning

By the end of this example, we’ve deployed a “playground” app where anyone with a browser can try out these custom models. That’s the power of Modal: custom, autoscaling AI applications, deployed in seconds. You can try out our deployment here.

Basic setup 

You will need to have an S3 bucket and AWS credentials to run this example. Refer to the documentation for the detailed IAM permissions those credentials will need.

After you are done creating a bucket and configuring IAM settings, you now need to create a Modal Secret. Navigate to the “Secrets” tab and click on the AWS card, then fill in the fields with the AWS key and secret created previously. Name the Secret s3-bucket-secret.

Modal runs serverless functions inside containers. The environments those functions run in are defined by the container Image. The line below constructs an image with the dependencies we need — no need to install them locally.

We attach the S3 bucket to all the Modal functions in this app by mounting it on the filesystem they see, passing a CloudBucketMount to the volumes dictionary argument. We can read and write to this mounted bucket (almost) as if it were a local directory.

For the base model, we’ll use a modal.Volume to store the Hugging Face cache.

Acquiring LoRA weights 

search_loras() will use the Hub API to search for LoRAs. We limit LoRAs to a maximum size to avoid downloading very large model weights. We went with 800 MiB, but feel free to adapt to what works best for you.

We want to take the LoRA weights we found and move them from Hugging Face onto S3, where they’ll be accessible, at short latency and high throughput, for our Modal functions. Downloading files in this mount will automatically upload files to S3. To speed things up, we will run this function in parallel using Modal’s map.

Inference with LoRAs 

We define a StableDiffusionLoRA class to organize our inference code. We load Stable Diffusion XL 1.0 as a base model, then, when doing inference, we load whichever LoRA the user specifies from the S3 bucket. For more on the decorators we use on the methods below to speed up building and booting, check out the container lifecycle hooks guide.

Try it locally! 

To use our inference code from our local command line, we add a local_entrypoint to our app. Run it using modal run cloud_bucket_mount_loras.py, and pass --help to see the available options.

The inference code will run on our machines, but the results will be available on yours.

LoRA Exploradora: A hosted Gradio interface 

Command line tools are cool, but we can do better! With the Gradio library by Hugging Face, we can create a simple web interface around our Python inference function, then use Modal to host it for anyone to try out.

To set up your own, run modal deploy cloud_bucket_mount_loras.py and navigate to the URL it prints out. If you’re playing with the code, use modal serve instead to see changes live.