Cloud bucket mounts

The modal.CloudBucketMount is a mutable volume that allows for both reading and writing files from a cloud bucket. It currently only supports AWS S3 buckets.

AWS S3 support is built on top of AWS’ mountpoint technology and inherits its limitations.

Mounting S3 buckets

CloudBucketMount enables S3 buckets to be mounted as file system volumes. To interact with a bucket, you must have the appropriate IAM permissions configured (refer to the section on IAM Permissions).

import modal
import subprocess

stub = modal.Stub()

s3_bucket_name = "s3-bucket-name"  # Bucket name not ARN.
s3_access_credentials = modal.Secret.from_dict({
    "AWS_ACCESS_KEY_ID": "...",
    "AWS_SECRET_ACCESS_KEY": "...",
})

@stub.function(
    volumes={
        "/my-mount": modal.CloudBucketMount(s3_bucket_name, secret=s3_access_credentials)
    }
)
def f():
    subprocess.run(["ls", "/my-mount"])

To mount a bucket in read-only mode, set read_only=True as an argument.

import modal
import subprocess

stub = modal.Stub()

s3_bucket_name = "s3-bucket-name"  # Bucket name not ARN.
s3_access_credentials = modal.Secret.from_dict({
    "AWS_ACCESS_KEY_ID": "...",
    "AWS_SECRET_ACCESS_KEY": "...",
})

@stub.function(
    volumes={
        "/my-mount": modal.CloudBucketMount(s3_bucket_name, secret=s3_access_credentials, read_only=True)
    }
)
def f():
    subprocess.run(["ls", "/my-mount"])

While S3 mounts supports both write and read operations, they are optimized for reading large files sequentially. Certain file operations, such as renaming files, are not supported. For a comprehensive list of supported operations, consult the Mountpoint documentation.

IAM permissions

To utilize CloudBucketMount for reading and writing files from S3 buckets, your IAM policy must include permissions for s3:PutObject, s3:AbortMultipartUpload, and s3:DeleteObject. These permissions are not required for mounts configured with read_only=True.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ModalBucketAccess",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::<MY-S3-BUCKET>"]
    },
    {
      "Sid": "ModalBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<MY-S3-BUCKET>/*"]
    }
  ]
}