Key Takeaways
- Serverless infrastructure can reduce operational overhead: Modal's code-first workflow lets teams express compute requirements in code and automatically scale workloads from zero to 1,000+ GPUs without directly managing VM capacity or Kubernetes for the core workflow. Modal also provides SDK interfaces in Python, JavaScript/TypeScript, and Go.
- GPU-specialized clouds emphasize accelerator infrastructure: CoreWeave focuses on GPU compute, networking, Kubernetes-based orchestration, and distributed training infrastructure.
- Hyperscalers combine GPU access with cloud ecosystems: AWS, Google Cloud, and Azure provide service portfolios alongside instance-based GPU infrastructure, managed ML services, scheduling, autoscaling, and capacity management.
- Bare-metal deployment is available for direct server access: Oracle Cloud Infrastructure offers bare-metal GPU systems alongside virtualized options for organizations that prefer direct hardware access.
- Regional deployment options vary by provider: Nebius operates GPU infrastructure across Europe, the US, and the Middle East, including European capacity relevant to data residency strategies.
- Modal combines security controls with serverless AI infrastructure: Modal has completed a SOC 2 Type II audit, uses gVisor-based workload isolation, and supports HIPAA-compliant workloads on Enterprise plans via a BAA.
1. Modal
Modal delivers serverless AI infrastructure for inference, training, batch processing, notebooks, and secure sandboxed execution across CPU and GPU compute. Its code-first workflow lets developers define compute requirements and execution behavior in code without operating Kubernetes for the core Modal workflow. Modal provides SDK interfaces through its Python SDK, JavaScript/TypeScript SDK, and Go SDK, including Sandbox operations across all three. Code running inside Modal Sandboxes is not limited to these SDK languages and can use the runtime or programming language the workload requires.
How Does Modal Work?
Modal packages application code into containers and executes it across a multi-cloud capacity pool. The platform automatically places workloads, manages compute provisioning, and scales demand-driven workloads from zero to 1,000+ GPUs.
Fast cold starts are a core part of the platform design. Modal is engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down.
Key capabilities include:
- GPU Portfolio: Modal GPU pricing lists B300, B200, H200 SXM, H100 SXM5, RTX PRO 6000, A100 80 GB, A100 40 GB, L40S, A10, L4, and T4 GPUs.
- Fast Cold Starts: Engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down.
- Memory Snapshotting: Memory Snapshots can checkpoint initialized application state so repeated startup work does not need to be performed from scratch.
- Per-Second Billing: Modal uses per-second billing for compute resources and can scale application compute down to zero.
- Multi-Cloud Capacity: Modal pools GPU capacity across multiple clouds and handles placement behind the serverless interface.
Enterprise Features
Modal's Enterprise plan includes enterprise controls such as audit logs and SSO. Modal has completed a SOC 2 Type II audit, uses gVisor-based isolation for compute workloads, and supports HIPAA-compliant workloads on Enterprise plans via a BAA.
Documented Results
Modal has reported thousands of customers and documents production use across generative AI inference, model training, robotics, media generation, coding agents, and large-scale batch workloads. Examples include:
- Decagon: The Decagon case study reports a 65% latency reduction after joint work on model training and inference optimization.
- Runway: The Runway case study describes real-time, multi-node inference for Runway Characters using distributed GPU execution and RDMA.
- Physical Intelligence: The Physical Intelligence case study documents remote inference for robotic control on Modal.
- Suno: The Suno case study reports that Suno brought its music generation model to market four months early instead of hiring a team to build and maintain the infrastructure.
- Ramp: The Ramp coding agent uses Modal Sandboxes for background coding workflows that generate code changes and write them back into commits or pull requests.
What Makes Modal Unique
Best For: Enterprise AI teams that want a fast path from code to production, elastic serverless scaling, modern GPU access, integrated AI infrastructure, and low infrastructure administration.
- Purpose-Built AI Infrastructure: Modal built core layers of its platform architecture, including its filesystem, container runtime, scheduler, and image builder, around compute-intensive workloads.
- Code-First Developer Experience: Modal provides SDK interfaces in Python, JavaScript/TypeScript, and Go, with code-defined workflows that let teams interact with Modal infrastructure while Modal handles provisioning, placement, autoscaling, and much of the infrastructure lifecycle.
- Flexible Execution: Modal supports both CPU and GPU workloads, while Sandboxes provide isolated environments that can run the runtime or language required by the application.
- Integrated Product Suite: Modal Inference supports model serving, Modal Training supports experiments and training, Modal Sandboxes provide isolated code execution, Modal Batch supports large-scale parallel processing, and Modal Notebooks provide CPU and GPU-backed interactive development.
2. CoreWeave
CoreWeave provides GPU-specialized cloud infrastructure for AI and machine learning workloads. Its platform is Kubernetes-native and is designed around dedicated accelerator infrastructure for training and inference.
Core Capabilities
- NVIDIA GPU infrastructure spanning Hopper and Blackwell generations.
- Networking for distributed GPU workloads.
- Managed Kubernetes through the CoreWeave Kubernetes Service.
- GPUDirect RDMA support for multi-node communication.
- Object storage, local NVMe, and additional storage options for AI data pipelines.
Enterprise Considerations
CoreWeave centers its operating model on cloud-native orchestration. Its managed Kubernetes and Slurm-based workflows support GPU-focused infrastructure through scheduling and cluster management workflows.
Best For: Organizations running large-scale distributed training or inference that prefer GPU-specialized infrastructure and Kubernetes-based orchestration.
3. AWS (Amazon EC2 Accelerated Computing)
AWS offers NVIDIA GPU instances and AWS-designed AI accelerators. Its GPU services integrate with AWS services for storage, networking, machine learning, security, data engineering, and application delivery.
GPU Instance Portfolio
- P6e-GB200: Blackwell-generation rack-scale GPU infrastructure.
- P6-B200 and P6-B300: Blackwell and Blackwell Ultra GPU configurations.
- P5 and P5e: H100 and H200 GPU instances.
- G7e: RTX PRO 6000 Blackwell Server Edition-based instances for accelerated workloads.
- Custom Silicon: Trainium and Inferentia options for workloads designed around AWS accelerators.
Enterprise Ecosystem
AWS combines GPU infrastructure with managed services such as SageMaker, Bedrock, EKS, storage systems, networking services, observability, identity, and security tooling. Enterprises already standardized on AWS can integrate GPU workloads into established governance and application architectures.
Scaling Options
AWS supports GPU clusters, EFA networking, managed ML infrastructure, reservation-based capacity options, and Spot capacity for interruption-tolerant workloads.
Best For: Enterprises with substantial AWS investments that want GPU access integrated with AWS cloud services and multiple infrastructure purchasing models.
4. Google Cloud Compute Engine GPUs
Google Cloud provides NVIDIA GPU compute through Compute Engine alongside Google-designed TPUs. This gives enterprise ML teams access to both NVIDIA accelerators and Google's TPU ecosystem within the same cloud platform.
GPU and Accelerator Options
- A4X and A4X Max: Blackwell-generation systems based on GB200 and GB300 configurations.
- A4: B200-based configurations for training and inference.
- A3 Ultra: H200-based systems for large AI workloads.
- A3 and A2: H100 and A100-based configurations.
- TPU7x Ironwood: Google's seventh-generation TPU platform for large-scale AI training and inference.
Platform Integration
Google Cloud combines accelerator infrastructure with managed ML lifecycle tooling, Google Kubernetes Engine, AI Hypercomputer, and Dynamic Workload Scheduler for capacity and scheduling workflows.
Cost Position
Google Cloud GPU pricing varies by accelerator, region, reservation structure, and purchasing model. The resulting economics depend on the specific workload and capacity configuration.
Best For: Organizations that value Google Cloud's TPU options, GKE, managed ML tooling, and integrated accelerator scheduling.
5. Microsoft Azure (ND-series)
Microsoft Azure provides enterprise GPU virtual machines through the ND-series and related accelerated compute families. Its portfolio includes Hopper and Blackwell-generation systems for large-scale training and inference.
GPU Infrastructure
- ND H200 v5: H200-based multi-GPU systems with GPU interconnect and InfiniBand networking.
- ND GB300 v6: Blackwell Ultra infrastructure for AI workloads.
- Additional accelerated VM families support a range of training, inference, graphics, and visualization use cases.
Microsoft Ecosystem Integration
Azure integrates GPU infrastructure with Microsoft Entra ID, Microsoft 365, Azure Arc, data services, networking, security tooling, and the broader Microsoft cloud ecosystem.
Azure Machine Learning
Azure Machine Learning provides managed MLOps capabilities for training, deployment, model management, evaluation, and monitoring within Azure environments.
Best For: Microsoft-centric enterprises that want GPU infrastructure integrated with Azure identity, governance, hybrid cloud, and managed ML services.
6. Oracle Cloud Infrastructure (OCI)
Oracle Cloud Infrastructure provides both bare-metal and virtualized GPU infrastructure. Its bare-metal model gives customers direct access to server hardware while retaining integration with OCI networking, storage, and cloud services.
Infrastructure Scale
OCI supports GPU clusters for distributed AI training and inference, including Blackwell-generation configurations connected through RDMA networking.
Bare Metal Deployment
Oracle's bare-metal GPU instances provide full server access for organizations that prefer dedicated hardware control and local storage configurations.
Deployment Options
Best For: Organizations that prioritize bare-metal GPU access, dedicated region deployment, data sovereignty options, or alignment with the Oracle ecosystem.
- OCI Dedicated Region: An Oracle-operated cloud region deployed in a customer-selected data center for sovereignty, residency, and isolation requirements.
- Customer-Owned GPU: Supported customer-owned GPU infrastructure can be deployed within Oracle dedicated cloud environments with Oracle-managed platform operations.
- Standard Cloud: Multi-tenant cloud GPU instances for conventional cloud deployment models.
7. Nebius
Nebius operates GPU-focused AI infrastructure across Europe, the US, and the Middle East. Its platform targets training, inference, and AI workloads with NVIDIA accelerators and cloud-native orchestration.
Infrastructure
- NVIDIA H100, H200, and B200 GPU infrastructure.
- InfiniBand networking for distributed GPU communication.
- Multi-region infrastructure with European capacity relevant to data residency requirements.
- Kubernetes and Slurm-based orchestration options.
Vertical Integration
Nebius designs elements of its servers and racks in-house and manages systems architecture, supply chain, deployment, and operations in collaboration with infrastructure partners.
Enterprise Features
The platform supports enterprise GPU infrastructure, managed services, Kubernetes, and Slurm workflows across its regional footprint.
Best For: Organizations seeking NVIDIA GPU infrastructure with European data residency options and cloud-native orchestration.
Why Modal Stands Out for Enterprise AI Infrastructure
Serverless Architecture Reduces Infrastructure Operations
Instance-based GPU clouds give infrastructure teams direct control over VM sizing, capacity, scheduling, and lifecycle choices, while hyperscalers also provide managed services that automate portions of those workflows. Modal's serverless model abstracts provisioning, container scheduling, GPU capacity management, and autoscaling behind code-defined application interfaces. Developers specify compute and execution behavior in code while Modal handles placement and scaling.
Purpose-Built for Modern AI Workloads
Modal engineered core layers of its stack for AI workloads, including its platform architecture, container runtime, scheduler, filesystem, and image builder. Fast cold starts are part of that design: Modal is engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down. Memory Snapshots can further reduce repeated initialization work by restoring initialized application state.
Complete Platform for the AI Lifecycle
Modal provides integrated products across development and production workflows:
- Inference: Real-time model serving with capabilities such as dynamic batching and snapshot-based startup optimization.
- Training: Single-node and multi-node training with GPU networking.
- Sandboxes: Isolated execution environments for AI agents, reinforcement learning rollouts, generated code, shell-based workflows, and other untrusted execution use cases.
- Batch Processing: Large-scale parallel processing for workloads with many independent inputs.
- Notebooks: Collaborative interactive development with CPU and GPU compute.
Enterprise-Grade Security and Compliance
Modal has completed a SOC 2 Type II audit and documents gVisor-based isolation for compute workloads and TLS 1.3 for public APIs. Modal supports HIPAA-compliant workloads on Enterprise plans via a BAA.
Proven at Scale
Modal has reported thousands of customers and documents production deployments across real-time model serving, robotics, audio generation, video generation, coding agents, biotech, training, and batch processing. Runway, Physical Intelligence, Suno, and Ramp provide concrete examples across different production AI patterns.
For enterprise AI teams evaluating GPU cloud options, Modal's combination of serverless developer experience, modern GPU access, multi-cloud capacity pooling, code-defined workflows, integrated inference and training tooling, secure sandboxed execution, and enterprise controls makes it the strongest choice in this guide for teams prioritizing development velocity and low infrastructure overhead. Additional production examples are available in Modal customer stories.
Explore Modal's AI infrastructure for enterprise workloads.
Explore Modal