Language model inference
“Modal's user-friendly interface and efficient tools have truly empowered our team to navigate data-intensive tasks with ease, enabling us to achieve our project goals more efficiently.”
“Switched to Modal for our LLM inference instead of Azure. 1/4 the price for GPUs and so much simpler to set up/scale. Big fan.”
“Using Modal for inference is like having an extra infra team - it’s reliable, scalable, and fast - meaning I can get back to training models”
GPUs on demand
Top of the line hardware
Access A100s and H100s to run the latest and largest models, like Llama3-405B.
Cheaper than running your own cluster
No more paying for idle GPUs.
Seamless autoscaling
When your app gets an influx of traffic, Modal scales with you.
Blazing-fast performance
Fast cold starts
Load gigabytes of weights in seconds with our optimized container file system and engine.
Support for inference engines
Easily run any framework or model on Modal (e.g. TensorRT and vLLM).
Dynamic batching
Use Modal's batching feature to process requests in dynamically-sized batches.
Best-in-class developer experience
Metrics and observability
Visualize and debug failures.
Monitor resource utilization
Track your usage and spending in real-time.
Ready for production
Support for webhooks, batching, and token streaming.
Try it out
Use Cases