Best Small Open Source Models for Cheap RL Rollouts in 2026

1. Modal

Modal provides serverless AI infrastructure for RL training, rollout inference, secure environment execution, batch processing, and interactive development, with compute available across CPUs and GPUs. Rather than managing clusters or reserving capacity, teams deploy jobs against Modal's multi-cloud GPU capacity pool, which is available without quotas or advance reservations.

Why Modal for RL Rollouts?

Modal combines the execution layer for both training and inference in a single platform:

• Fast cold starts: Engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down. Modal documents the full cold start path, including queueing and first-invocation initialization such as weight loading, library imports, kernel compilation, engine startup, and cache warmup, so teams can plan readiness end to end.
• Elastic autoscaling with scale-to-zero: Burst from zero to more than 1,000 GPUs without capacity planning or reservations. Modal describes scaling times ranging from seconds for certain clusters to minutes at 1,000-GPU scale, with scheduling shaped by GPU type, count, topology, region, and available capacity.
• Native sandbox support: Run untrusted code from RL agents in secure sandboxes built for the high-volume environment execution that RL infrastructure demands. Sandbox workloads are not tied to a single language: the sandbox runs whatever runtime the environment requires.
• Code-first developer experience: Define infrastructure in code using Modal SDKs in Python, TypeScript, and Go, so rollout environments, training jobs, and inference endpoints live alongside the RL loop itself.
• Multi-node training: Connect GPU clusters over a 3,200 Gbps RoCE RDMA scale-out network with a 50 Gbps private network and support for up to 64 devices. Modal's first-party pages currently describe this capability as Beta.

Documented Performance

Modal's infrastructure has measurable results in production AI workloads:

• Physical Intelligence reports 10 to 15 ms of added network overhead for real-time remote robotic control inference
• Decagon Voice 2.0 achieved a 65% latency reduction following joint model-training and inference-engine optimization work by Decagon and Modal, spanning supervised fine-tuning and RL, compact open source models, custom draft models, and inference-engine tuning
• Ramp runs background coding agents on Modal Sandboxes that generate code changes and write them back into commits and pull requests
• The platform powers cloud infrastructure for more than 10,000 teams, including Runway and Quora

Security and Compliance

Modal states that it has successfully completed a SOC 2 Type II audit and is SOC 2 Type II compliant. Modal supports HIPAA-compliant workloads on Enterprise plans via a BAA. Compute jobs run containerized and virtualized using gVisor, which adds a user-space kernel boundary around workload syscalls, with user data encrypted in transit and at rest and public APIs using TLS 1.3, per Modal's security documentation. Best For: Teams that want elastic GPU capacity for RL training without capacity management overhead, and that benefit from training, inference, and sandboxed execution living on one platform.

2. Qwen3-4B

Qwen3-4B is a strong candidate at the top of the small-model range for single-GPU RL fine-tuning work. The official card reports 4.0B total parameters and 3.6B non-embedding parameters; calculating from the published architecture and tensor dimensions gives approximately 4.02 billion parameters in a dense architecture. A note on hardware: parameter-efficient fine-tuning fits comfortably on a single 80GB GPU. Full-parameter mixed-precision AdamW training does not fit comfortably. At roughly 18 bytes per parameter for weights, gradients, and optimizer state, about 72.4GB is consumed before activations, CUDA context, and temporary tensors, leaving little headroom. Fitting it requires aggressive optimization such as optimizer-state quantization, activation checkpointing or offload, very short sequences, a microbatch size of one, or LoRA/QLoRA instead of full-parameter training. An RL workflow may additionally need policy, reference, reward, value, or rollout-serving components depending on the algorithm.

Key Capabilities

• Dual reasoning modes: Toggle between thinking and non-thinking modes at inference time within a single checkpoint, without retraining
• Extended context: Native 32,768 token context window, with YaRN validation reported up to 131,072 tokens
• Tool integration: Native tool-calling support, with Qwen recommending its Qwen-Agent tooling for agentic RL applications
• License: Apache 2.0, subject to the license's notice, attribution, trademark, and patent conditions

Benchmark Performance

Results are labeled by evaluation mode, because thinking and non-thinking scores are not comparable. The figures below come from the comparison tables published in the SmolLM3 model card, which evaluates Qwen3 checkpoints in both modes alongside its own:

Thinking mode more than triples the AIME 2025 result (approximately 244% relative) and more than doubles LiveCodeBench (approximately 112% relative), while GSM-Plus moves by roughly 7% relative. The gain is real but highly benchmark-dependent, and it is paid for in generated tokens: the Qwen3 card recommends output allowances as large as 38,912 tokens for difficult math and programming evaluations. Best For: RL training that needs stronger mathematical and coding reasoning and can accommodate parameter-efficient fine-tuning on a single GPU. Run Qwen3-4B inference on Modal's GPU instances, selecting the GPU type and count that fit your rollout profile.

3. SmolLM3-3B

SmolLM3-3B offers one of the more transparent training recipes among competitive 3B models. Hugging Face published the training data mixture, code, configs, and intermediate checkpoints, which lets researchers inspect much of how the model reaches its performance. That disclosure is extensive but not yet complete: the model card states that the datasets used for mid-training and post-training will be uploaded later.

Architecture Highlights

• Context: Configured for 65,536 tokens, which is twice Qwen3-4B's native 32,768-token window. The card's headline capability describes training at 64K with support up to approximately 128K using YaRN; the supplied configuration example uses factor 2, producing 131,072 tokens. The long-context section mentions 128k or 256k, but the card does not provide equivalent validation evidence for 256K, so treat roughly 128K as the documented extended window.
• Training scale: 11.2 trillion tokens with a staged data curriculum
• Architecture: GQA with NoPE (no positional embeddings) in a documented 3:1 pattern for efficient long-context processing
• Post-training: Reasoning mid-training, supervised fine-tuning, and Anchored Preference Optimization
• License: Apache 2.0

Performance Metrics

Mode-labeled results from the official evaluation tables:

On instruction following and tool use, the card reports 76.7 on IFEval and 92.3 on BFCL tool calling, both no-thinking results. Best For: Research teams that want a largely reproducible baseline with published pretraining data and intermediate checkpoints, while accounting for the pending mid-training and post-training data release. Deploy SmolLM3-3B through Modal's batch processing for large-scale rollouts.

4. Qwen3-1.7B

Qwen3-1.7B is a practical entry point for rapid RL prototyping. The official card uses the 1.7B label; calculating from the published configuration gives approximately 1.72 billion parameters.

Efficiency Advantages

• Comparison to Qwen2.5-3B: At roughly 43% fewer labeled parameters, the Qwen3 technical report finds that Qwen3-1.7B-Base outperforms Qwen2.5-3B-Base on more than half of the evaluated base-model benchmarks, particularly in STEM and coding. The instruction-model comparison is mixed: in no-thinking mode, Qwen3-1.7B is better on some tasks and worse on others.
• Memory footprint: Approximately 31GB for mixed-precision AdamW model states (1.72B parameters at roughly 18 bytes each), before activations, temporary tensors, and CUDA overhead
• Iteration speed: Lower model-state memory and computation generally allow faster training cycles at a fixed configuration
• Device flexibility: Quantized versions are available for local inference, though actual laptop compatibility depends on RAM, quantization, context length, and runtime

RL-Relevant Benchmarks

Mode-labeled results from the comparison tables in the SmolLM3 model card, which evaluates Qwen3 checkpoints in both modes:

Best For: Iterating on RL algorithms where training velocity matters more than maximum capability. Train Qwen3-1.7B on Modal with autoscaling tuned to your workload, using warm pools, buffers, and scaledown windows to match standing capacity to rollout demand.

5. OLMo-1B-0724-hf

OLMo-1B from Ai2 (formerly styled the Allen Institute for AI) prioritizes scientific reproducibility. The training data, training and evaluation code, intermediate checkpoints, and logs are publicly released, which lets researchers study training dynamics directly rather than inferring them. Two qualifications matter for this list. First, OLMo-1B-0724-hf is a base pretrained model, not an instruction-tuned checkpoint, so its out-of-the-box rollout behavior is not directly comparable to the instruct models elsewhere in this guide. Second, it is an older option: Ai2 now offers OLMo 2 1B as a newer small model and describes OLMo 3 as its latest family. The 0724 checkpoint remains appropriate where checkpoint density and exact historical reproducibility matter, but a current 1B shortlist should also evaluate OLMo 2 1B.

Transparency Features

• Complete disclosure: Training data, code, and process documented and released
• Checkpoint releases: Multiple intermediate training states available for analysis
• Academic backing: Maintained by Ai2
• Architecture: Decoder-only transformer with non-parametric LayerNorm, SwiGLU, no bias terms, and rotary positional embeddings. RMSNorm was considered but not selected in the original OLMo; RMSNorm and QK-Norm were introduced later, in OLMo 2.

Research Applications

The published configuration specifies 16 layers, hidden size 2,048, intermediate size 8,192, a 50,304-token vocabulary, and untied input/output embeddings, implying approximately 1.28 billion actual parameters despite the 1B label. With Apache 2.0 licensing, it serves academic teams that need:

• Reproducible RL training baselines
• Training dynamics analysis through intermediate checkpoints
• Building on fully documented model architectures

Best For: Academic and research RL work requiring complete reproducibility and training transparency, where a base model is acceptable. Run OLMo experiments on Modal notebooks for interactive GPU-backed development.

6. Qwen3-0.6B

Qwen3-0.6B is the smallest entry in the Qwen3 lineup covered here and still retains thinking mode. The official rounded size is 0.6B; the published configuration implies approximately 596 million parameters.

Compact Efficiency

• Model-state memory: Approximately 10.7 to 11GB for mixed-precision AdamW weights, gradients, and optimizer state, before activations, temporary tensors, CUDA overhead, and RL-specific auxiliary components
• Full feature set: Dual reasoning modes despite the small size
• Context: 32,768 token native context architecture
• Edge deployment: Quantized versions can be run through local inference runtimes, subject to device memory and performance constraints

Performance Characteristics

Qwen3-0.6B is best treated as a fast iteration target rather than a capability play. Published comparable figures at this size are sparse: the Qwen3 technical report reports 24.74 on MMLU-Pro for the Qwen3-0.6B base model, which is a base-model result and not a thinking versus non-thinking comparison. If you need thinking-mode numbers at this size, generate them yourself under a stated harness, prompt, shot count, answer-extraction method, and checkpoint revision, since the official card does not publish a traceable thinking-mode IFEval or MMLU-Pro pair. Best For: Maximum iteration speed during RL algorithm development and educational use. Deploy through Modal's inference platform for production serving, where fast cold starts keep the iteration loop tight.

7. SmolLM2-360M-Instruct

SmolLM2-360M-Instruct sits at the bottom of the size range covered in this guide. At 360 million parameters, it supports CPU inference and beats a larger comparator on several benchmarks, though not on all of them. Whether it is sufficient for a given RL task is an empirical question for your own reward signal, not a property of the size tier.

Efficient Training Foundation

• Training corpus: The base model was trained on 4 trillion tokens
• Post-training: Supervised fine-tuning followed by DPO with UltraFeedback
• Resource efficiency: The model card provides CPU execution examples
• License: Apache 2.0
• Data caveat: The SmolLM2 card notes that some newly curated filtered datasets will be released later

Benchmark Comparison

Under the model card's evaluation setup, the instruct checkpoint scores:

• 41.0 on IFEval, versus 31.6 for Qwen2.5-0.5B-Instruct
• 52.1 on HellaSwag
• 43.7 on ARC (average)

Note that the frequently quoted 54.5 HellaSwag and 53.0 ARC figures belong to the base SmolLM2-360M model, not the instruct checkpoint. Against Qwen2.5-0.5B-Instruct, SmolLM2-360M-Instruct is ahead on IFEval, HellaSwag, ARC, PIQA, and MMLU, and behind on MT-Bench, BBH, and GSM8K. It outperforms the larger comparator on several evaluations, not generally. Best For: Algorithm prototyping and educational RL work where compute cost must be minimized. Scale SmolLM2-360M rollouts through Modal's job queue, which supports autoscaling, retries, and up to one million pending asynchronous inputs.

Why Modal Stands Out for RL Rollouts

Purpose-Built Infrastructure for AI Workloads

Modal's serverless platform removes much of the infrastructure work that slows RL experimentation. The platform provides GPU access without quotas or reservations across a multi-cloud capacity pool, so teams can size rollouts to experiment demand rather than to pre-provisioned capacity. Modal describes bursting to more than 1,000 GPUs with headline timing measured in minutes, and seconds for certain clusters, with allocation shaped by hardware, topology, region, and supply.

Unified Training and Inference Platform

Modal combines model training, inference serving, sandboxed execution, batch processing, and notebooks on a single platform, defined in code through SDKs in Python, TypeScript, and Go. RL workflows that run model inference, execute environment code, persist state, and train policy updates can use these integrated services and primitives, which reduces the amount of external infrastructure a team must operate.

Fast Cold Starts Support Cost Efficiency

Modal is engineered for fast cold starts and faster feedback loops, with an optimized filesystem that helps containers come online quickly without letting large images slow startup down, and Functions scale to zero by default when no inputs remain. For bursty rollout patterns this can meaningfully reduce idle spend, while min_containers, buffer_containers, and scaledown_window give teams direct control over how much warm capacity stays available between bursts. Modal documents the full cold start path, including queueing and first-invocation initialization, and Memory Snapshots can accelerate initialization-heavy Functions, often by 3 to 10 times according to Modal, with GPU Memory Snapshots extending the same approach to GPU state as an Alpha feature.

Native Support for RL Primitives

The platform includes primitives that map onto common RL patterns:

• Queues: Coordinate asynchronous and distributed rollout work
• Volumes: Persist model checkpoints and replay data; Modal's official GRPO example defines a Volume specifically for checkpoints, and its long-running training example saves state to persistent storage so interrupted jobs can resume
• Sandboxes: Execute untrusted agent code in isolated environments with configurable network controls, including the option to block outbound network access, and run whatever language or runtime the environment requires
• Snapshots: Filesystem, directory, and memory snapshots restore sandbox state quickly instead of rebuilding from scratch, with memory snapshots currently in Alpha. Directory snapshots can capture just the project-owned portion of a sandbox and be mounted after startup, which suits patterns such as attaching project-specific state to pre-warmed sandboxes

Enterprise-Grade Security

Modal states that it has completed a SOC 2 Type II audit, with compute isolation through gVisor containerization and virtualization and encryption in transit and at rest, per its security documentation. gVisor places a user-space kernel between workload syscalls and the host, which is what makes running LLM-generated code in a sandbox practical at scale. Modal supports HIPAA-compliant workloads on Enterprise plans via a BAA.

Proven at Scale

Modal reports powering infrastructure for more than 10,000 teams. Physical Intelligence reports 10 to 15 ms of added network overhead for remote robotic control inference, Runway uses Modal for real-time multi-node inference for Runway Characters, its real-time video-agent API, and Quora uses Modal Sandboxes at scale to run LLM-generated code for Poe. On the agent side, Ramp built a full-context background coding agent on Modal Sandboxes that generates code changes and writes them back into commits and pull requests. Modal's suitability for reinforcement learning is supported further by its RL infrastructure write-up, the Applied Compute RL case study, official GRPO implementations with TRL and verl, and customers such as Cognition that report using Modal for both RL infrastructure and production inference. For teams running RL rollouts with small open source models, Modal's combination of elastic GPU scaling, per-second billing without mandatory reservations, and integrated platform capabilities makes it a strong infrastructure choice for experimentation. Explore Modal's GRPO example to see implementation patterns for GRPO and other RL algorithms.