Preemption
All Modal functions are subject to preemption. If a preemption event interrupts a running function, Modal will gracefully terminate the function and restart it on the same input.
Preemptions are rare, but it is always possible that your function is interrupted. Long-running functions such as model training functions should take particular care tolerate interruptions as likelihood of interruption increases with function runtime.
Preparing for interruptions
Design your applications to be fault and preemption tolerant. Modal will send an
interrupt signal (SIGINT) to your application when preemption occurs. In
Python applications, this signal is by default propogated as a
KeyboardInterrupt, which you can handle in your code to perform cleanup.
Other best practices for handling preemptions include:
- Divide long-running operations into small tasks or use checkpoints so that you can save your work frequently.
- Ensure preemptible operations are safely retryable (ie. idempotent).
Running uninterruptible functions
We currently don’t have a way for functions to avoid the possibility of interruption, but it’s a planned feature. If you require functions guaranteed to run without interruption, please reach out!