Modal logo
IntroductionCustom container images Defining ImagesUsing existing container imagesFast pull from registryGPUs and other resources GPU accelerationUsing CUDA on ModalReserving CPU and memoryScaling out Scaling outInput concurrencyBatch processingJob queuesDynamic batching (beta)Multi-node clusters (beta)Deployment Apps, Functions, and entrypointsManaging deploymentsInvoking deployed functionsContinuous deploymentRunning untrusted code in FunctionsModal Sandboxes SandboxesRunning commandsNetworking and securityFile accessSnapshotsModal NotebooksSecrets and environment variables SecretsEnvironment variablesScheduling and cron jobsWeb endpoints Web endpointsStreaming endpointsWeb endpoint URLsRequest timeoutsProxy Auth TokensNetworking TunnelsProxies (beta)Cluster networkingData sharing and storage Passing local dataVolumesStoring model weightsCloud bucket mountsDictsQueuesDataset ingestionPerformance Cold start performanceMemory SnapshotHigh-performance LLM inferenceGeographic latencyReliability and robustness Failures and retriesPreemptionTimeoutsGPU healthTroubleshootingSecurity and privacyIntegrations Using OIDC to authenticate with external servicesConnecting Modal to your Datadog accountConnecting Modal to your OpenTelemetry providerOkta SSOCustom SAML SSOSlack notifications (beta)Workspace & account settings WorkspacesEnvironmentsModal user account setupService usersBillingOther topics JavaScript/Go SDKsModal 1.0 migration guideFile and project structureDeveloping and debuggingDeveloping Modal code with LLMsJupyter notebooksAsynchronous API usageGlobal variablesRegion selectionContainer lifecycle hooksParametrized functionsS3 Gateway endpointsGPU Metrics

Streaming endpoints

Modal fastapi_endpoints support streaming responses using FastAPI’s StreamingResponse class. This class accepts asynchronous generators, synchronous generators, or any Python object that implements the iterator protocol, and can be used with Modal Functions!

Simple example 

This simple example combines Modal’s @modal.fastapi_endpoint decorator with a StreamingResponse object to produce a real-time SSE response.

If you serve this web endpoint and hit it with curl, you will see the ten SSE events progressively appear in your terminal over a ~5 second period.

The MIME type of text/event-stream is important in this example, as it tells the downstream web server to return responses immediately, rather than buffering them in byte chunks (which is more efficient for compression).

You can still return other content types like large files in streams, but they are not guaranteed to arrive as real-time events.

Streaming responses with .remote 

A Modal Function wrapping a generator function body can have its response passed directly into a StreamingResponse. This is particularly useful if you want to do some GPU processing in one Modal Function that is called by a CPU-based web endpoint Modal Function.

Streaming responses with .map and .starmap 

You can also combine Modal Function parallelization with streaming responses, enabling applications to service a request by farming out to dozens of containers and iteratively returning result chunks to the client.

This snippet will spread the ten map_me(i) executions across containers, and return each string response part as it completes. By default the results will be ordered, but if this isn’t necessary pass order_outputs=False as keyword argument to the .map call.

Asynchronous streaming 

The example above uses a synchronous generator, which automatically runs on its own thread, but in asynchronous applications, a loop over a .map or .starmap call can block the event loop. This will stop the StreamingResponse from returning response parts iteratively to the client.

To avoid this, you can use the .aio() method to convert a synchronous .map into its async version. Also, other blocking calls should be offloaded to a separate thread with asyncio.to_thread(). For example:

Further examples