modal.gpu

GPU configuration shortcodes

The following are the valid str values for the gpu parameter of @stub.function.

“t4” → GPU(T4, count=1)
“l4” → GPU(L4, count=1)
“a100” → GPU(A100-40GB, count=1)
“a10g” → GPU(A10G, count=1)
“inf2” → GPU(INFERENTIA2, count=1)
“any” → GPU(Any, count=1)

Other configurations can be created using the constructors documented below.

modal.gpu.A100

class A100(modal.gpu._GPUConfig)

NVIDIA A100 Tensor Core GPU class.

The most powerful GPU available in the cloud. Available in 40GiB and 80GiB GPU memory configurations.

def __init__(
    self,
    *,
    count: int = 1,  # Number of GPUs per container. Defaults to 1. Useful if you have very large models that don't fit on a single GPU.
    memory: int = 0,  # Set this to 80 if you want to use the 80GB version. Otherwise defaults to 40.
):

modal.gpu.A10G

class A10G(modal.gpu._GPUConfig)

NVIDIA A10G Tensor Core GPU class.

A10G GPUs deliver up to 3.3x better ML training performance, 3x better ML inference performance, and 3x better graphics performance, in comparison to NVIDIA T4 GPUs.

def __init__(
    self,
    *,
    count: int = 1,  # Number of GPUs per container. Defaults to 1. Useful if you have very large models that don't fit on a single GPU.
):

modal.gpu.Any

class Any(modal.gpu._GPUConfig)

Selects any one of the GPU classes available within Modal, according to availability.

def __init__(self, *, count: int = 1):

modal.gpu.L4

class L4(modal.gpu._GPUConfig)

NVIDIA L4 GPU class.

Mid-tier GPU option, providing 24GiB of GPU memory.

def __init__(
    self,
    count: int = 1,  # Number of GPUs per container. Defaults to 1. Useful if you have very large models that don't fit on a single GPU.
):

modal.gpu.T4

class T4(modal.gpu._GPUConfig)

NVIDIA T4 GPU class.

Low-cost GPU option, providing 16GiB of GPU memory.

def __init__(
    self,
    count: int = 1,  # Number of GPUs per container. Defaults to 1. Useful if you have very large models that don't fit on a single GPU.
):