“With the old in-house systems, we'd have to tune number of workers, instance size, parallelization strategy, all this stuff, which was very time-consuming and not directly generating business value. Modal magically handled all that.”
“Processing external quantum mechanical datasets comes with unique challenges. Jobs can fail in numerous ways—from low-level errors thrown by the underlying analysis packages, to transient issues communicating with our storage server. Modal's retry mechanism and batching primitives have made our data pipeline much more robust.”
Define your task with Modal’s SDK. You can specify environment and hardware requirements in line with your application code.
image = (
modal.Image.debian_slim().pip_install("numpy")
)
@app.function(
image=image,
volumes={"/data": volume},
)
def build_year_wrapped(user_id):
import numpy as np
...
Call the function on your batch of inputs. You can queue up to 1M inputs.
build_year_wrapped.spawn_map
(user_id for user_id in range(1_000_000))
Modal instantly launches thousands of containers and reliably processes inputs to completion.
Drill down into logs and metrics for individual inputs.
Zoom out to quickly see aggregate metrics for each batch.
Easily configure Slack or email notifications for failures