I am often asked “What is Modal?” People want a simple, quick, intuitive answer, so I end up reaching for analogies to other products and services. I don’t like most of them.

• “Modal is Uber for compute” — but 1) it’s not 2016 anymore and 2) we don’t operate a two-sided open market.
• “Modal is a neocloud” — but we don’t operate hardware and you don’t rent machines directly.
• “Modal is an inference provider” — but we don’t sell tokens, we sell processor-seconds and byte-seconds you can use to provide inference services. We're closer to an “inference provider provider”.
• “Modal is AI infrastructure” — sure, it’s great for that, but it can do much more!

What I wish I could say instead: “Modal is a computer."

What is a computer?

A computer is a machine that runs programs of arithmetic & logic operations on information. A typical computer looks something like this:

Information lives in a tiered hierarchy of storage, ranging from registers inside its arithmetic/logic unit (ALU) to blocks in a filesystem on its disk. The key level is the memory, which can hold onto information as long as the computer has power. Any information that wants to come in from the outside world or go out to it needs to go through the memory first, via memory-mapped input-output (I/O). Getting information from/to other computersin a network is an important special case, and that information passes through a network interface card (NIC).

What is Modal?

Modal is a machine that runs programs of arithmetic & logic operations on information. A computer!

It looks something like this:

Information lives in a tiered hierarchy of storage, ranging from memory inside of user containers to items in object storage. The key level is the memory of the container runtime, which can hold onto information as long as the worker is running. Any information that wants to come in from the outside world or go out to it needs to go through that memory, via network communication with our Input/Output Plane. Getting information from/to other computers in a network is an important special case, and that information passes through our Routing Plane or via direct Tunnels.

How do programs run on Modal and other computers?

Programs run cores and containers. In a typical computer, the ALU fetches inputs and instructions from memory and applies them to operands in registers. Those programs might be stateless tools that consume stdin and produce stdout, like most “Unix-y” tools. Or they might be generic Linux processes plus a permissions model, like most containers. Or they might specifically be Linux processes that listen for network requests.

In Modal, containers receive inputs from the container runtime and apply their program to them. Those programs might be stateless Python functions that consume inputs and produce outputs, as in Modal Functions. Or they might be generic Linux processes plus a permissions model, as in Modal Sandboxes. Or they might specifically be Linux processes that listen for HTTP requests, as in Modal Servers.

Those instructions need to be placed in memory by a loader, usually part of the computer’s operating system, which manages, virtualizes, and time-shares all the resources of the computer, providing the efficiency of multitenancy with the security of single tenancy.

The containers need to be placed on a machine by part of the container runtime, which manages, virtualizes, and time-shares all the cloud resources we operate, providing the efficiency of multitenancy with the security of single tenancy.

How do programs end up inside of Modal and other computers?

Programs are generally stored in a "compiled" state different from their definition format and then pulled in for execution.

Instructions in the memory of a typical computer must generally first be compiled to object code from their source code — once, then stored on disk for retrieval by the operating system when the program is turned into a process and run. Many program instances use the same, shared object code, like glibc.so, so the operating system arranges for that data to be transparently re-used across them. Among the most critical of those shared components are the drivers, which communicate, via memory, with devices. These are handled specially by the operating system.

In Modal, container images must generally first be “compiled” to a filesystem from their definition — once, and then placed in object storage for retrieval by the container runtime when the Image is turned into a container and run. Many containers use the same files, like glibc.so or PyTorch, so the container runtime arranges for that data to be transparently re-used across them. Among the most critical of those shared components are the drivers, which communicate, via the container runtime, with devices. These are handled specially by the container runtime.

Where is information stored in Modal and other computers?

The outermost level of storage handles long-term persistence of information.In a typical computer, information can only be directly operated on inside the ALU when stored in registers. But registers are small relative to our programs’ ambitions, so most information lives, for up to the lifetime of a program, in memory. Excess data is pushed to disk. To provide something between the performance of all information living in registers and the capacity of all information living on disk, the hardware and operating system offer a hierarchy of caches. The innermost level of these caches includes the L0/L1 CPU cache and TLB, which are transparently managed by the hardware. GPUs also offer a programmer-managed scratchpad, the shared memory of SMs. The outermost level of these caches is the filesystem. Programmers can place additional information on other systems, e.g. in databases or object storage.

In Modal, information can only be directly operated on inside a container when stored in memory. But memory is small relative to our containers’ ambitions, so most information lives, for up to the lifetime of the program, on disk. Excess data is pushed to object storage. To provide something between the performance of all information living in memory and the capacity of all information living in object storage, the container runtime offers a hierarchy of caches. The innermost level of these caches is the page cache, which is transparently managed by the container runtime. Modal also offers a programmer-managed scratchpad, the ephemeral_disk of the machine. The outermost level of these caches is comprised of Modal Volumes. Programmers can place additional information on other systems, e.g. in databases or object storage.

How does information get into or out of Modal and other computers?

Information passes from interfaces through the central (ephemeral) storage component. In a typical computer, there are two primary channels for input/output of information: the keyboard+mouse/screen for information provided by/to humans; and the network, for information provided by/to other computers. These passthrough routing logic and memory buffers in the operating system to reach running programs. In case of extreme performance need, these can be partially bypassed, directly connecting remote programs with ours via remote direct memory access, which communicates via InfiniBand Verbs (usually).

In Modal, there are two primary channels for input/output of information: the Modal Input/Output Plane for information provided by/to callersof Modal Functions or executors of Modal Sandboxes; and the Modal Routing Plane, for information provided via HTTP by/to Modal Servers. These passthrough routing logic and storage buffers in Modal to reach running containers. In case of extreme performance need, these can be partially bypassed, directly connecting remote programs with containers via Modal Tunnels, which communicate via TCP streams or UDP datagrams.

Why use Modal if it’s “just” a computer? Aren’t the other computers good enough?

Operating systems construct an essentially virtual machine out of the raw hardware on which they run. Processes virtualize CPU time; memory virtualizes RAM; filesystems virtualize the disk; the network stack virtualizes bandwidth. This virtualization allows the operating system to aggregate, isolate, and multiplex demand onto that hardware for increased resource efficiency.

Modal is very much a virtual computer in this same vein. But instead of operating on the raw hardware in a single machine, it operates on all the hardware offered by dozens of cloud providers. And by building another layer of virtualization, we can aggregate, isolate, and multiplex a higher abstraction of demand onto that hardware for increased resource efficiency at a grander scale. Pretty important engineering task for a time where compute resources are tight!

Individual clouds do something like this too, but they are, in our opinion, far too low level for productive work by most engineering teams. At least, they felt that way when we were working on other applications and looking at peers’ work, so we decided to just build the thing we thought everyone needed instead.

If you’d like to help us build that computer too, we’re hiring.