Sandbox Secret Injection

An Outbound Policy lets a Sandbox call external HTTPS APIs that require authentication without the API key ever being visible inside the Sandbox.

Sandboxes are often used to run untrusted code. Passing API keys to such code as environment variables is risky: any process in the Sandbox can read its own environment and exfiltrate the keys. An Outbound Policy avoids this by declaring headers replacements which are injected into outbound HTTPS requests to given domains. This replacement happens outside the Sandbox so the secret never appears in the Sandbox’s environment.

For a complete working demonstration, see the Outbound Policy example.

Defining an Outbound Policy 

An Outbound Policy is an immutable configuration object built by chaining with_header_replacement calls. Each call adds a domain, a set of headers to inject into requests to that domain, and optionally a Secret whose keys can be referenced in the header values.

import modal
import modal.experimental

secret = modal.Secret.from_name("api-token")

outbound_policy = (
    modal.experimental.OutboundPolicy()
    # Inject a secret-backed Authorization header.
    .with_header_replacement(
        domain="example.com",
        secret=secret,
        headers={"Authorization": "Bearer $API_TOKEN"},
    )
    # Inject a static header into requests to another domain.
    .with_header_replacement(
        domain="modal.com",
        headers={"X-Trace-Token": "trace_abcd"},
    )
)

import { ExperimentalOutboundPolicy, ModalClient } from "modal";

const modal = new ModalClient();
const secret = await modal.secrets.fromName("api-token");

const outboundPolicy = new ExperimentalOutboundPolicy()
  // Inject a secret-backed Authorization header.
  .withHeaderReplacement({
    domain: "example.com",
    secret,
    headers: { Authorization: "Bearer $API_TOKEN" },
  })
  // Inject a static header into requests to another domain.
  .withHeaderReplacement({
    domain: "modal.com",
    headers: { "X-Trace-Token": "trace_abcd" },
  });

secret, err := mc.Secrets.FromName(ctx, "api-token", nil)
if err != nil {
	return err
}

outboundPolicy := new(modal.ExperimentalOutboundPolicy).
	// Inject a secret-backed Authorization header.
	WithHeaderReplacement(modal.ExperimentalOutboundPolicyHeaderReplacement{
		Domain:  "example.com",
		Secret:  secret,
		Headers: map[string]string{"Authorization": "Bearer $API_TOKEN"},
	}).
	// Inject a static header into requests to another domain.
	WithHeaderReplacement(modal.ExperimentalOutboundPolicyHeaderReplacement{
		Domain:  "modal.com",
		Headers: map[string]string{"X-Trace-Token": "trace_abcd"},
	})

A Outbound Policy can contain up to 25 header values across all header replacement added to it.

Domain matching 

Each header replacement’s domain supports exact hostnames, *. wildcard prefixes, and a bare "*":

Multiple replacement entries may cover the same domain. All matching replacements apply in the order they were added. An injected header replaces any header with the same name that existed on the request, either set by the Sandbox workload or any previously matched replacements.

Header value templates 

Header values support templating against the keys of a supplied Secret. A $-prefixed key name is replaced with the secret value at request time. To write a literal $, escape it as $$.

A replacement without a secret is static, and its header values are used verbatim when injected into the request.

Attaching a policy to a Sandbox 

Pass the policy to Sandbox.create:

sb = modal.Sandbox.create(
    "python", "agent.py",
    app=app,
    _experimental_outbound_policy=outbound_policy,
)

const sb = await modal.sandboxes.create(app, image, {
  command: ["python", "agent.py"],
  experimentalOutboundPolicy: outboundPolicy,
});

sb, err := mc.Sandboxes.Create(ctx, app, image, &modal.SandboxCreateParams{
	Command:        []string{"python", "agent.py"},
	ExperimentalOutboundPolicy: &outboundPolicy,
})

This policy is invisible from the Sandbox: none of the secrets used in any header replacements are added to the Sandbox environment.

Updating the policy on a running Sandbox 

You can replace the Outbound Policy on a running Sandbox without restarting it. Updating the policy will fully replace the existing one previously configured.

sb._experimental_update_outbound_policy(
    outbound_policy.with_header_replacement(
        domain="*",
        headers={"X-Sandbox-Id": sb.object_id},
    )
)

await sb.experimentalUpdateOutboundPolicy(
  outboundPolicy.withHeaderReplacement({
    domain: "*",
    headers: { "X-Sandbox-Id": sb.sandboxId },
  }),
);

updated := outboundPolicy.WithHeaderReplacement(modal.ExperimentalOutboundPolicyHeaderReplacement{
	Domain:  "*",
	Headers: map[string]string{"X-Sandbox-Id": sb.SandboxID},
})
err := sb.ExperimentalUpdateOutboundPolicy(ctx, &updated)
if err != nil {
	return err
}

Updating the policy only works if the Sandbox was created with an Outbound Policy. If you want to update the policy later you can create a Sandbox with an empty Outbound Policy.

How it works 

When a Sandbox is created with an Outbound Policy, Modal attaches a small proxy that sits on the Sandbox’s outbound network path.

Outbound TCP traffic on port 443 is redirected through the proxy, which reads the destination domain from the SNI in the TLS handshake.

If the domain matches a policy rule, the proxy terminates the TLS connection using a certificate minted by certificate authority (CA) unique to the Sandbox. The CA’s certificate is installed into the Sandbox’s trust store at creation time, see TLS trust inside the Sandbox. The proxy injects the configured headers and forwards the request to the destination over a separate TLS connection.

If the domain does not match any rule, the connection is passed through untouched.

Only HTTPS on port 443 is eligible for injection. Plain HTTP, other ports, and non-TLS protocols are never terminated or modified. Connections using Encrypted Client Hello (ECH) hide their real destination from the proxy and are passed through without injection.

TLS trust inside the Sandbox 

TLS termination only works if the workload trusts the Certificate Authority used in the proxy. At creation time Modal installs the CA certificate in the root filesystem at /etc/modal/sandbox-egress-ca.crt. We also try to append this to any existing system CA bundle the image has if we can recognize it.

Modal also sets common environment variables in the Sandbox pointing towards a CA bundle containing the Sandbox CA certificate:

• SSL_CERT_FILE
• REQUESTS_CA_BUNDLE
• CURL_CA_BUNDLE
• NODE_EXTRA_CA_CERTS

If your chosen HTTP client does not recognize these environment variables, or if your system CA bundle is in a location where we don’t append the CA certificate, you will need to manually configure the client to read the certificate from /etc/modal/sandbox-egress-ca.crt.

Limitations 

There are some known limitations to this feature. We are working on improving this feature, but for now we are aware that the following does not work:

• We only support HTTP/1.1 in both directions. The proxy installed will only advertise HTTP/1.1 over ALPN and will only use HTTP/1.1 when forwarding the connection to the destination. This means that HTTP/2-only traffic, including gRPC, will not work.
• Only named Secrets will work. Ephemeral Secrets (Secret.from_dict) are not supported yet.
• An Outbound Policy cannot be combined with block_network, outbound_domain_allowlist, outbound_cidr_allowlist, the experimental proxy_traffic_via_sidecar option, or memory snapshots (enable_snapshot).
• An Outbound Policy can only be applied to the main Sandbox container. Sidecar containers are not covered.
• Secret values are resolved when the Sandbox is created and on each _experimental_update_outbound_policy call. A running Sandbox will not pick up new values if the Secret is updated after this.