Network file systems (superseded)

Modal lets you create writeable volumes that can be simultaneously attached to multiple Modal Functions. These are helpful for use cases such as:

1. Storing datasets
2. Keeping a shared cache for expensive computations
3. Leveraging POSIX filesystem APIs for both local and remote data storage

Note: NetworkFileSystem has been superseded.

The NetworkFileSystem abstraction is limited by the fact that the underlying storage is located in only one cloud region. Since Modal compute runs in multiple regions, this causes variable latency and throughput issues when accessing the file system.

To address this, we have a new distributed storage primitive, modal.Volume, that offers fast reads and writes across all regions. NetworkFileSystems are still supported and useful in some circumstances, but we recommend trying out Volumes first for most new projects.

Basic example

New modal.NetworkFileSystem objects can be created through the modal CLI:

modal nfs create

or they can be created on-the-fly when using the modal.NetworkFileSystem.from_name constructor with create_if_missing=True.

In your App code, the filesystem can be mounted within a Function by providing a mapping between mount paths and NetworkFileSystem objects. For example, to use a NetworkFileSystem to initialize a shared shelve disk cache:

import shelve
import modal

volume = modal.NetworkFileSystem.from_name("my-cache", create_if_missing=True)

@app.function(network_file_systems={"/root/cache": volume})
def expensive_computation(key: str):
    with shelve.open("/root/cache/shelve") as cache:
        cached_val = cache.get(key)

    if cached_val is not None:
        return cached_val

    # cache miss; populate value
    ...

The above implements basic disk caching, but be aware that shelve does not guarantee correctness in the event of concurrent read/write operations. To protect against concurrent write conflicts, the flufl.lock package is useful.

Deleting NFS objects

You can delete a network filesystem object (along with all of its data) via the storage dashboard, the modal nfs delete CLI, or the modal.NetworkFileSystem.delete method.