A note on scope: this guide uses "environment" as an umbrella term. It compares environment-generation systems, evaluation harnesses, environment libraries, and RL training frameworks used to build and train terminal agents, since teams typically assemble a stack from more than one of these categories. We start with Modal's AI infrastructure platform, which provides the serverless execution, secure sandboxing, and compute across CPUs and GPUs that RL rollout collection and policy training both demand.
Key Takeaways
- Infrastructure quality shapes training throughput: Terminal agent RL requires rapid environment spin-up, secure isolation, and compute across CPUs and GPUs. Modal delivers fast cold starts and supports 100,000+ concurrent sandboxes with fast Sandbox scheduling, making it a strong foundation for scalable agent training.
- Harbor is the official Terminal-Bench 2.0 harness: The Harbor Framework comes from the creators of Terminal-Bench and is widely integrated across current terminal-agent work, with roughly 4,000 GitHub stars as of August 2026 and 50+ documented benchmark adapters.
- Synthetic environment generation scales RL training: Research from Endless Terminals reports absolute held-out development-set gains of 14.2 to 42.6 percentage points across three evaluated models, providing evidence that scalable, diverse environments can drive substantial gains even with a comparatively simple RL setup.
- Composable architectures ease production pipelines: The verifiers library's taskset × harness × runtime design allows teams to mix task sets with execution harnesses and run them through distinct runtimes, including local subprocess and Docker execution.
- Security and compliance matter for enterprise adoption: Modal completed a SOC 2 Type 2 audit, and its Sandboxes use gVisor to provide strong isolation and malicious-syscall protection for untrusted, agent-generated code. Modal supports HIPAA-compliant workloads on Enterprise plans via a BAA.
1. Modal
Modal provides the serverless compute infrastructure for running terminal agent workloads at scale. The company built a custom file system, container runtime, scheduler, and image builder rather than assembling general-purpose cloud primitives, and its optimized filesystem is tuned so containers come online quickly. The result is fast cold starts across inference, training, batch processing, and Sandboxes, with a container stack that brings GPUs online for Functions quickly.
Key Capabilities
- Secure sandbox execution: Modal Sandboxes are purpose-built for executing untrusted user or AI-generated code, and are built on gVisor, which provides strong isolation properties and custom logic against malicious system calls, limiting the blast radius of malicious code to the Sandbox container. This matters directly when training agents that write and run arbitrary shell commands.
- Fast cold starts: Engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down.
- Code-first SDKs across languages: Sandboxes are created and controlled through Modal's code-first SDKs, with sandbox operations available in Python, TypeScript, JavaScript, and Go. Code running inside a Sandbox is not limited to a single language either, so a terminal agent can invoke whatever runtime or toolchain a task requires.
- Configurable network and resource boundaries: Sandboxes are isolated from workspace resources and reject unsolicited incoming connections by default, and teams that want to close off outbound traffic can run a Sandbox with
block_network=True. Resource boundaries are equally configurable: you can pin CPU, memory, and other resource ceilings for misbehaving or adversarial workloads, tune the requested amounts, and allow bursting above them where headroom helps. - CPU and GPU compute: Workloads run across CPU and GPU resources, with direct access to A100, H100, H200, B200, B300, L40S, L4, A10, T4, and RTX PRO 6000 GPUs, automatic container-pool autoscaling, and hardware pooled across multiple clouds.
- Snapshotting and warm-pool patterns: Modal supports filesystem, directory, and memory snapshots for Sandboxes. Filesystem snapshots can initialize new Sandboxes from saved state, which is useful for curriculum checkpoints and failure recovery. Directory snapshots capture only part of a Sandbox, so user or project files can be preserved separately from platform-owned dependencies, and because they can be mounted after a Sandbox has started, they pair naturally with a common latency-optimization pattern: maintaining a warm pool of pre-started Sandboxes that launch a server, pull a repo, and install dependencies before an end user is ever waiting. Memory snapshots are available as an Alpha feature with documented usage guidance.
- Serving and connectivity: A server running inside a Sandbox can be exposed through a Modal tunnel URL, and connection tokens authenticate access to that server, which is the pattern to reach for when sandbox-backed previews or environments are embedded directly in an application. Files reach a Sandbox by being baked into images, through volumes, through mounted cloud buckets, or through the filesystem API.
- Production-grade reliability and governance: Modal has successfully completed a SOC 2 Type 2 audit and supports HIPAA-compliant workloads on Enterprise plans via a BAA.
Why Modal Ranks First
Terminal agent RL is an infrastructure problem before it is an algorithms problem: rollout collection needs thousands of short-lived, isolated shell environments launched on demand, and policy updates need GPUs immediately afterward. Modal addresses both halves natively. Its Sandboxes product explicitly positions itself for RL rollouts, and Modal's own engineering writing on reinforcement learning as an infrastructure problem works through the scaling behavior that RL training loops demand.
Modal also supports both of the architecture patterns agent teams actually use. Running the agent inside the Sandbox is the easier place to start and is common for internal coding agents, while running the agent outside the Sandbox gives cleaner separation of concerns and is the pattern that platforms with proprietary agent logic tend to prefer, and which looks like the long-term direction for the category. Teams can begin with whichever fits today and move between the two as their product matures, without leaving the platform.
The production track record covers precisely the latency-sensitive and agentic workloads that terminal RL teams graduate into. Physical Intelligence runs real-time remote inference for robotic control on Modal with roughly 10 to 15 ms of cloud-network overhead, and Ramp built a full-context background coding agent on Modal Sandboxes, generating code changes and writing them back into commits and pull requests. For terminal agent RL specifically, the combination of strong sandbox isolation, fast cold starts, multi-cloud CPU and GPU capacity, and per-second billing creates a foundation that carries a project from research experimentation through production deployment.
Best For: Teams requiring production-grade infrastructure for terminal agent training and deployment, particularly those needing secure sandbox execution for untrusted code at high concurrency.
2. Harbor Framework
Harbor Framework is the official evaluation harness for Terminal-Bench 2.0 and a widely integrated framework for terminal-agent evaluation and RL workflows. Created by the Terminal-Bench team, it enables reproducible evaluation across the 89 Terminal-Bench 2.0 tasks listed on Harbor Hub. Terminal-Bench 2.1 describes itself as a more verified iteration of 2.0, modifying 26 tasks to address bugs, resource and timeout issues, and reward-hacking robustness.
Key Capabilities
- Broad agent support: Evaluate Claude Code, OpenHands, Codex, Aider, Goose, Gemini CLI, and many additional installed agents through a unified interface
- Benchmark integration: The repository documents 50+ supported benchmark adapters, including SWE-Bench-family datasets, Aider Polyglot, and LiveCodeBench, while Harbor Hub currently lists 262 datasets (dataset count is broader than unique benchmark families, but it illustrates the size of the registry)
- Cloud execution backends: The repository lists Daytona, E2B, and GKE among supported execution environments for parallel evaluation
- RL rollout generation: Dedicated workflows for generating training data from containerized evaluations
- Multi-agent trajectories: The Agent Trajectory Interchange Format explicitly supports subagent delegation and hierarchical multi-agent architectures
Harbor's roughly 4,000 GitHub stars as of August 2026 and Apache-2.0 license have driven wide adoption. SETA explicitly uses Harbor as its unified task format, Endless Terminals converted a large portion of its generated tasks to Harbor format, and verifiers ships a built-in Harbor taskset, so Harbor functions in practice as a shared interchange format across several current projects.
Best For: Research teams and enterprises requiring standardized, reproducible evaluation of terminal agents across many benchmarks.
3. Prime Intellect verifiers
The verifiers library from Prime Intellect provides a composable architecture for creating RL environments and evaluations, with roughly 4,500 GitHub stars as of August 2026. Prime Intellect reports thousands of hosted training runs across its broader RL stack, and says the new verifiers abstractions have been used for Prime's production training and evaluation needs.
Key Capabilities
- Composable design: The taskset × harness architecture allows any task set to run under any compatible execution harness
- Unified runtime abstraction: Runtime is a distinct abstraction from the taskset and harness, with local execution via subprocess or Docker containers
- First-class branching: Support for non-linear rollouts enabling compaction and subagent coordination, which the project says makes training across subagents feasible
- Training-ready traces: Message graphs with token IDs and log probabilities for faithful RL training
- Harbor interoperability: A built-in Harbor taskset, described as the first fully supported third-party taskset format, makes porting Harbor datasets straightforward
On July 10, 2026, Prime Intellect announced verifiers 0.2.0, which previews the rewritten v1 core under the verifiers.v1 namespace and includes a "Towards 1.0.0" section outlining the remaining roadmap. The repository is MIT-licensed.
Best For: Production teams building end-to-end training pipelines that need flexible environment composition, and that separately need scalable execution infrastructure for rollout collection; Modal's batch processing supports queues of up to one million inputs.
4. Endless Terminals
Endless Terminals provides a fully autonomous pipeline for generating terminal RL tasks at scale. The Stanford and Microsoft Research collaboration produced 3,255 tasks covering file operations, log management, data processing, scripting, and database operations, with an open-source codebase.
Key Capabilities
- Four-stage generation pipeline: Automated description generation, container validation, test generation, and solvability filtering
- Substantial training gains: On the held-out development set, Llama-3.2-3B improved from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%, gains of 14.2, 42.6, and 16.4 percentage points respectively
- Transfer to benchmarks: Training gains transfer to Terminal-Bench 2.0, with Qwen2.5-7B improving from 2.2% to 3.4%
- Harbor format output: The 3,255 tasks were generated in Apptainer format, and approximately 2,500 of them were also converted to Harbor format for standardized evaluation; the paper's own experiments used the Apptainer pipeline
Training used vanilla PPO with binary episode rewards and a deliberately minimal scaffold, which is the most interesting part of the result: substantial gains were possible in this setup without retrieval, multi-agent scaffolding, or specialized agent tools. That provides evidence that scaling automatically generated environments can drive agent capability even when the RL setup stays simple, rather than proving a general causal law about environment diversity versus algorithmic sophistication.
Best For: Research teams investigating environment scaling hypotheses and teams needing large-scale synthetic training data for terminal agents.
5. SETA
SETA from CAMEL-AI is a synthetic terminal environment generation framework paired with an opinionated agent harness. It debuted in January 2026 and was substantially extended by a July 12, 2026 paper, which should be treated as the current reference: SETA-Env now contains 4,567 tasks across 14 tech-stack categories, well beyond the initial snapshot.
Key Capabilities
- Robust terminal toolkit: The current TerminalToolkit exposes seven methods:
shell_exec, shell_view, shell_wait, shell_write_to_process, shell_kill_process, shell_write_content_to_file, and shell_ask_user_for_help. The last is disabled during headless training, leaving six operational methods in that setting. - Note-taking system: Persistent memory for long-horizon tasks spanning dozens of steps
- Dual-agent synthesis: Automated task generation using Idea and Datapoint agent stages in SETA-Synth
- Denser reward signal: SETA adopts partial-progress rewards based on unit-test pass fraction rather than relying purely on sparse binary episode rewards
- RL training results: The best Qwen3-8B SETA RL run reached 12% pass@1 on Terminal-Bench 2.0 versus 3.6% for the best base run, a 3.3× improvement that the authors report as the strongest result for an RL-trained model at the 8B scale
One point of context on a figure that circulates widely: the January CAMEL harness result of 46.5% on Terminal-Bench 2.0 with Claude Sonnet 4.5 was ranked first at the time of publication, and leaderboard standings have since moved as the field has advanced. That harness score is also a different experiment from SETA's RL-trained model results above. The codebase is Apache-2.0 licensed.
Best For: Teams focused on agent harness and toolkit design, and those wanting a large, verified synthetic task pool with denser reward shaping.
6. SkyRL
SkyRL describes itself as a modular full-stack RL library for LLMs, and it is the framework used in the Endless Terminals research. It spans more than a training loop: the repository contains skyrl-train for the training stack, skyrl-agent for long-horizon agents, and skyrl-gym, a Gymnasium-based environment library that supplies tool-use environments. For a terminal-agent stack, skyrl-gym is the component that maps most directly onto the "environment" category.
Key Capabilities
- Full-stack modularity: Separable training, agent, and environment layers that can be adopted independently
- PPO implementation: A configurable PPO stack that Endless Terminals used for its vanilla-PPO training runs
- Environment library:
skyrl-gym provides Gymnasium-style tool-use environments for agentic training - Research-proven results: Used in published research achieving training improvements across models from 3B to 8B parameters, and SkyRL lists Endless Terminals among the projects built on it
One clarification on figures that are often attributed to SkyRL itself: the specific clipping bounds, sequence-level loss averaging, five-minute environment timeout, 16-turn training limit, 64-turn evaluation limit, and 4× A100 or 8× B200 hardware configurations come from the Endless Terminals experiment implemented using SkyRL. They are study-specific settings, not SkyRL defaults or architectural constraints.
Best For: Research teams conducting RL algorithm experiments who need a modular, proven full-stack library with a first-party environment layer.
7. OpenClaw-RL
OpenClaw-RL is a fully asynchronous RL framework, announced in v1 form on February 26, 2026, that continuously converts eligible live interaction turns and subsequent feedback into training samples. This agent-native approach differs from traditional offline batch RL collection.
Key Capabilities
- Asynchronous architecture: Serving, rollout collection, judging, and training are separated, allowing policy optimization to proceed while the agent continues serving requests
- Eligible-turn training signal: OpenClaw-RL classifies messages into main-line trainable turns and side non-trainable turns; in the Binary RL path, a final turn that never receives a next state is normally excluded from training, except for an at-least-one-turn fallback, so not every interaction becomes a training sample
- Continuous in-use optimization: The framework can update an agent's policy during use, allowing deployment interactions to contribute to ongoing optimization
- Broad agent coverage: The general-agent track spans terminal, GUI, SWE, and tool-call scenarios
The continuous learning approach reduces dependence on a fixed offline train-then-deploy cycle by allowing policy updates from live interaction data. It does not eliminate training, and the published evidence supports asynchronous continual optimization from live next-state feedback rather than a general guarantee that agents adapt to arbitrary new environments without further training. The repository is Apache-2.0 licensed.
Best For: Teams building production agents that must keep adapting during use, and those exploring online RL paradigms.
Why Modal Stands Out for Terminal Agent RL
Purpose-Built Infrastructure for AI Workloads
Modal's custom file system, container runtime, scheduler, and image builder are engineered specifically for AI workloads, and its optimized filesystem helps containers come online quickly without letting large images slow startup down. Modal is tuned for the rapid container spin-up and the CPU and GPU access patterns that terminal agent RL demands, delivering fast cold starts and fast Sandbox scheduling. For curriculum learning and failure recovery, filesystem and directory snapshots can capture and restore Sandbox state, and because directory snapshots can be mounted after a Sandbox has started, project-specific state can be attached to pre-warmed capacity at the moment a rollout begins.
Enterprise-Grade Security for Untrusted Execution
Terminal agents generate and execute arbitrary shell commands, which makes the execution boundary central to the design. Modal Sandboxes are built on gVisor, providing strong isolation properties and custom logic against malicious system calls, with the blast radius of malicious code limited to the Sandbox container. Teams that want a full Linux kernel underneath their workloads can also use Modal's VM Sandboxes runtime.
Two design levers are worth knowing. First, network posture is configurable: Sandboxes reject unsolicited incoming connections by default, and outbound traffic can be closed off with block_network=True where preventing data movement is a requirement. When a Sandbox does need to serve traffic, a server inside it can be exposed through a Modal tunnel, with connection tokens authenticating access, and tunnel traffic connects directly to the Sandbox rather than routing through Modal's control plane, so for an EU-scheduled Sandbox, data does not leave the EU through that control plane apart from traffic between a client and a server outside the region. Region pinning is a guarantee rather than a preference: a Sandbox is scheduled in the chosen region or not at all. Second, on compliance: Modal has completed a SOC 2 Type 2 audit and supports HIPAA-compliant workloads on Enterprise plans via a BAA.
Seamless Scaling From Research to Production
Modal Environments let the same App be deployed into development and production environments without changing code, and Modal's Hello World guide demonstrates running the same function locally, remotely, and remotely in parallel. Teams can therefore develop locally, test at small scale, and deploy to production without rewriting their execution path, using code-defined infrastructure in Python, TypeScript, or Go.
On raw scale, the current Sandboxes product page documents 100k+ concurrent sandboxes. Beyond that, Modal reported on July 16, 2026 that it ran one million Sandboxes concurrently and created all one million in under a minute on its redesigned Sandbox system, described in the V2 Sandbox documentation. That result indicates the ceiling the architecture is being built toward for massively parallel rollout collection.
Integrated Observability and Collaboration
Modal Notebooks provide collaborative development environments backed by CPU and GPU compute, with the ability to switch to up to eight H100s or B200s and real-time collaboration through multiple cursors, live edits, and shared context. Production deployments benefit from built-in logs, metrics, and resource-utilization visibility, with logs available for inspection in the Modal dashboard and for export to external tooling including Datadog. Governance features scale with the plan: deployment rollbacks and role-based access control are available on Team and Enterprise, and audit logs on Enterprise. Modal's pricing page lays out the tier boundaries.
