Network file systems
Modal lets you create writeable volumes that can be simultaneously attached to multiple Modal functions. These are helpful for use cases such as:
- Storing datasets
- Keeping a shared cache for expensive computations
- Leveraging POSIX filesystem APIs for both local and remote data storage
Basic example
The modal.NetworkFileSystem
constructor initializes an empty volume. This can be mounted within a function
by providing a mapping between mount paths and NetworkFileSystem
objects. For
example, to use a NetworkFileSystem
to initialize a shared
shelve disk cache:
import shelve
import modal
volume = modal.NetworkFileSystem.new()
@stub.function(network_file_systems={"/root/cache": volume})
def expensive_computation(key: str):
with shelve.open("/root/cache/shelve") as cache:
cached_val = cache.get(key)
if cached_val is not None:
return cached_val
# cache miss; populate value
...
The above implements basic disk caching, but be aware that shelve
does not
guarantee correctness
in the event of concurrent read/write operations. To protect against concurrent
write conflicts, the flufl.lock
package is useful. An example of that library’s usage is in the
Datasette example.
Persisting volumes
By default, a modal.NetworkFileSystem
lives as long as the app it’s defined in, just like any other Modal object.
However in many situations you might want to persist file data between runs of
the app. To do this, you can use the persisted
method on the
NetworkFileSystem
object. For example, to durably store trained model
checkpoints when running a model training job:
import modal
volume = modal.NetworkFileSystem.persisted("job-storage-vol")
stub = modal.Stub()
MODEL_DIR = "/models"
@stub.function(
network_file_systems={MODEL_DIR: volume},
)
def run_training():
...
...
trainer.save(MODEL_DIR)
Deleting volumes
To remove a persisted network file system, deleting all its data, you must “stop” it. This can be done via the network file system’s dashboard app page or the CLI.
For example, a file system with the name my-vol
that lives in the e-corp
workspace could be stopped (i.e. deleted) by going to its dashboard page at
https://modal.com/apps/e-corp/my-vol and clicking the trash icon. Alternatively,
you can use the file system’s app ID with
modal app stop
.
(Network File Systems are currently a specialized app type within Modal, which is why deleting one is done by stopping an app.)
Further examples
- The Modal Podcast Transcriber uses a persisted network file system to durably store raw audio, metadata, and finished transcriptions.