Run massively parallel evals on Modal Sandboxes

Modal Sandboxes are useful for executing code generated by a language model, creating isolated environments for running untrusted code, and more. Here we use Harbor, a framework for evaluating and optimizing agents and models in container environments, to run massively parallel evals. Harbor creates the Modal Sandboxes via harbor run --env modal.

Set up

This script requires some dependencies to be installed locally. We include the following inline script metadata so that tools like uv can automatically install these dependencies.

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "harbor[modal]==0.15.0",
#   "modal~=1.5.3",
# ]
# ///

To run this example:

uv run --python 3.12 --script 13_sandboxes/harbor_evals.py

import argparse
import os
import subprocess
import sys
import tempfile
from pathlib import Path

Define the task

A Harbor task is a directory of files that Harbor uses to set up the environment, instruct the agent, and verify its results. Harbor runs each attempt at a task as a trial in its own Modal Sandbox. The default task is a toy example to demonstrate basic functionality.

DEFAULT_TASK_FILES = {
    "task.toml": """version = "1.0"
[environment]
docker_image = "ubuntu:24.04"
workdir = "/app"
network_mode = "no-network"
cpus = 1
memory_mb = 512
storage_mb = 1024
""",
    "instruction.md": "Create `answer.txt` containing exactly `ok`.\n",
    "solution/solve.sh": "#!/bin/bash\nprintf 'ok\\n' > answer.txt\n",
    "tests/test.sh": """#!/bin/bash
if [ "$(cat /app/answer.txt 2>/dev/null)" = "ok" ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
  exit 1
fi
""",
}

Run the evals

We simply write the task files to a temporary directory, then let Harbor take care of the rest.

def main() -> None:
    parser = argparse.ArgumentParser(
        description="Run parallel Harbor oracle evals on Modal Sandboxes."
    )
    parser.add_argument("--n-parallel", type=int, default=8)
    parser.add_argument(
        "--task",
        type=Path,
        default=None,
        help="Harbor task directory.",
    )
    args = parser.parse_args()

    has_toml = (Path.home() / ".modal.toml").exists()
    has_env = bool(
        os.environ.get("MODAL_TOKEN_ID") and os.environ.get("MODAL_TOKEN_SECRET")
    )
    if not has_toml and not has_env:
        print(
            "Modal auth required: run `modal token new` or set "
            "MODAL_TOKEN_ID and MODAL_TOKEN_SECRET.",
            file=sys.stderr,
        )
        raise SystemExit(1)

    with tempfile.TemporaryDirectory() as tmp:
        run_dir = Path(tmp) / os.urandom(16).hex()
        run_dir.mkdir(parents=True, exist_ok=True)
        jobs = run_dir / "jobs"

        if args.task is None:
            task = run_dir / "task"
            (task / "environment").mkdir(parents=True)
            for name, contents in DEFAULT_TASK_FILES.items():
                path = task / name
                path.parent.mkdir(parents=True, exist_ok=True)
                path.write_text(contents)
                if name.endswith(".sh"):
                    path.chmod(0o755)
        else:
            task = args.task.expanduser().resolve()
            if not task.is_dir():
                print(f"Task path must be a directory: {task}", file=sys.stderr)
                raise SystemExit(1)

        print(
            f"Running {args.n_parallel} trials on {args.n_parallel} concurrent "
            "Modal Sandboxes...",
            flush=True,
        )
        subprocess.run(
            [
                "harbor",
                "run",
                "--path",
                str(task),
                "--agent",
                "oracle",
                "--env",
                "modal",
                "--n-attempts",
                str(args.n_parallel),
                "--n-concurrent",
                str(args.n_parallel),
                "--jobs-dir",
                str(jobs),
                "--yes",
                "--quiet",
            ],
            check=True,
        )
        rewards = [path.read_text().strip() for path in jobs.glob("**/reward.txt")]
        passed = sum(reward == "1" for reward in rewards)
        total = args.n_parallel

    print(f"{passed}/{total} tests passed")
    if passed != total:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Next steps

The default of eight parallel Sandboxes finishes quickly and fits within every Modal plan. To see real scale, pass a larger value, up to your plan’s concurrent container limit:

uv run --python 3.12 --script 13_sandboxes/harbor_evals.py --n-parallel 100

You can also pass a local directory to run your own task instead:

uv run --python 3.12 --script 13_sandboxes/harbor_evals.py --task ./my-task

Run massively parallel evals on Modal Sandboxes

Set up

Define the task

Run the evals

Next steps

Try this on Modal!