RLVR
Reinforcement learning with verifiable rewards (RLVR) depends on reliable signals for mathematical reasoning, code correctness, and step-level progress. This guide compares six leading open-source verifier models and the Modal infrastructure teams use to train, serve, and scale complete RLVR workflows.
Verifier models turn candidate answers, reasoning traces, or generated programs into training signals. In RLVR, those signals can come from outcome verification, process-level judgments, generated critiques, code execution, or learned estimates of correctness. The models below span each approach. Reported benchmark results are tied to their original evaluation settings and should be validated on the policy, task distribution, and reward pipeline used in production.
Modal is a serverless AI infrastructure platform rather than a verifier checkpoint. It ranks first because a production RLVR system needs more than a model: it needs scalable policy and verifier inference, GPU training, distributed rollouts, data processing, and secure execution for generated code. Best For: Teams building end-to-end RLVR pipelines that need GPU flexibility, burst scaling, and isolated code execution without operating Kubernetes clusters.
Modal powers cloud infrastructure for over 10,000 teams. RLVR workloads can move from local code to distributed training and serving while retaining the same application definitions. Memory Snapshots, an optimized filesystem, autoscaling, and high-concurrency Sandboxes help reduce the operational overhead of bursty rollout and verification stages. Actual startup time and throughput depend on image size, model loading, hardware, region, and workload behavior.
Modal Sandboxes use gVisor-based isolation to reduce host-kernel exposure when RLVR tasks execute untrusted generated code. Modal has completed a SOC 2 Type II audit with no deviations found and supports HIPAA-compliant workloads on Enterprise plans through a BAA. Teams remain responsible for reward design, secrets handling, network policy, and validating verifier outputs.
CompassVerifier-7B is a Qwen2-based verifier from OpenCompass designed for unified answer verification and outcome reward. Its training work aggregates more than one million samples from responses produced by over 50 models across 15 datasets, targeting cases where rules and general-purpose LLM judges fail. Best For: General-purpose outcome verification across mixed reasoning domains and teams that want one verifier interface for evaluation and RL rewards.
The CompassVerifier paper and repository report F1 and accuracy on VerifierBench and compare the 7B checkpoint with specialized verifiers including xVerify and Tencent's Qwen2.5-7B-Instruct-RLVR. The model is positioned as a strong size-to-quality choice within the CompassVerifier family. Because VerifierBench combines heterogeneous datasets and judgment types, teams should reproduce evaluation on their own answer formats and error distribution.
Qwen2.5-7B-Instruct-RLVR comes from the work Expanding RL with Verifiable Rewards Across Diverse Domains. It applies RLVR beyond a single narrow benchmark and uses the Qwen2.5-7B-Instruct foundation, making it relevant both as an RLVR-trained policy and as a reference for reward design across verifiable tasks. Best For: Researchers studying cross-domain RLVR and teams already standardized on Qwen2.5 tooling, tokenization, and serving infrastructure.
This checkpoint is not a drop-in replacement for a deterministic checker. Teams should define whether they need a policy model, a judge, or a separate reward model and evaluate calibration before using its outputs as rewards. Full-precision weights need substantially more memory than quantized serving, while production throughput varies with sequence length, batching, attention implementation, and GPU choice.
ThinkPRM-14B is a generative process reward model based on DeepSeek-R1-Distill-Qwen-14B. Rather than assigning a step score without explanation, it generates a verification chain of thought and then judges correctness. The project trains its verifiers with 1,000 synthetic verification traces filtered using 8,000 process labels from PRM800K. Best For: Math-reasoning pipelines that need interpretable step-level feedback and want to scale verifier compute by sampling more or longer verification traces.
ThinkPRM treats verification as a reasoning task. For each candidate step, the model can inspect the preceding solution, generate a detailed critique, double-check its analysis, and produce a correctness judgment. This makes reward failures inspectable and allows test-time scaling in parallel through multiple samples or sequentially by allocating more thinking tokens. Generated rationales can still be persuasive and wrong, so they require benchmark and downstream validation.
In the ThinkPRM paper, ThinkPRM-14B outperforms a discriminative PRM built on the same base model on ProcessBench while using far fewer supervision signals. It also shows stronger Best-of-N accuracy scaling than the compared LLM-as-a-judge, discriminative PRM, and majority-vote baselines on the paper's AIME 2024 and MATH-500 settings. Those curves depend on the generator, sampling count, and evaluation setup.
GenPRM-7B is a generative process reward model built on DeepSeek-R1-Distill-Qwen-7B. It reasons about each solution step before predicting whether that step is correct, and its training method combines chain-of-thought verification with code-assisted checks and Relative Progress Estimation. Best For: Mathematical reasoning systems that need a compact generative verifier, code-assisted process supervision, or a critic model for iterative policy refinement.
GenPRM supplements natural-language verification with code verification when a step can be checked computationally. The project uses Relative Progress Estimation to improve process labels and a rationale-synthesis pipeline to produce supervision for explicit verification reasoning. At inference time, teams can sample multiple verification traces and aggregate judgments, trading additional compute for potentially stronger reliability.
The GenPRM paper reports that its models outperform prior PRMs using roughly 23,000 MATH training examples. With test-time scaling, GenPRM-1.5B outperforms GPT-4o and GenPRM-7B surpasses Qwen2.5-Math-PRM-72B on ProcessBench in the authors' setup. These are process-verification results, not general claims that the smaller models outperform those systems on every reasoning task.
Granite-3.3-8B-Math-PRM-v2 is IBM Research's Apache-2.0-licensed process reward model for mathematical reasoning. Released in January 2026, it scores intermediate steps and supports inference scaling on both math and code benchmarks. Best For: Teams that need an enterprise-friendly, permissively licensed 8B PRM for step-level math verification and Best-of-N selection.
An 8B checkpoint is practical on a single modern accelerator when loaded at an appropriate precision, but exact memory use depends on weights, runtime, context length, KV cache, and batch size. Quantization can lower the footprint at the cost of a possible ranking-quality change. For RLVR, teams should test step-score calibration, aggregation strategy, and downstream policy improvement rather than relying only on standalone benchmark accuracy.
CodeScaler-8B is an MIT-licensed reward model for code generation. It learns from preference data derived from verified programming problems and uses syntax-aware code extraction plus validity-preserving reward shaping. Its central goal is to replace repeated test execution with learned scoring during reward-model inference. Best For: Code RL pipelines that need high-throughput candidate scoring, lack tests for synthetic problems, or want to combine a learned verifier with selective execution-based validation.
CodeScaler scores code from the problem and candidate text without running a unit-test suite for every reward-model inference. That can reduce latency and makes reward assignment possible for synthetic tasks without tests. Execution-free scoring remains a learned estimate rather than proof of correctness; robust systems should retain held-out execution, adversarial evaluation, and calibration checks.
The CodeScaler paper reports that reward-model training improves Qwen3-8B-Base by an average of 11.72 points across five coding benchmarks in one reported evaluation, outperforming binary execution-based RL by 1.82 points. A later paper revision reports gains across four benchmarks and up to 14.64 points after scaling training to 44,000 problems without requiring test cases. These figures reflect the authors' specific training recipes and benchmark suites.
Open-source verifiers provide reward signals; Modal provides the operating layer that turns those signals into repeatable training and production systems.
Modal's custom runtime, scheduler, image builder, and filesystem are designed for AI workloads that load large models, use heterogeneous accelerators, and fan out many short-lived tasks. Policy inference, verifier inference, and execution workers can each use different resources without sharing a fixed cluster.
Use Modal Training for policy or verifier fine-tuning, Modal Inference for model serving, Modal Batch for rollout and scoring jobs, and Modal Sandboxes for code-backed rewards. A shared code-first platform reduces handoffs between data generation, verification, training, and evaluation.
Modal supports over 10,000 teams and documents 100,000+ concurrent Sandboxes on its current product page. Modal has also demonstrated one million concurrent Sandboxes created in under a minute. Those demonstrations show platform capacity; application throughput remains dependent on quotas, images, regions, models, and task behavior.
RLVR often executes code that is incorrect, adversarial, or simply unpredictable. Modal combines gVisor isolation, secrets management, network controls, resource limits, and timeouts with SOC 2 Type II controls and support for HIPAA-compliant workloads on Enterprise plans through a BAA. This lets teams isolate execution without separating it from the rest of the RL pipeline.
Build verifier inference, parallel rollouts, RL training, and isolated execution on one serverless platform.
Explore Modal DocsA verifier model converts a candidate answer, reasoning step, or generated program into a reward signal. RL training uses that signal to increase the probability of outputs judged correct or useful. Outcome verifiers score completed responses, while process reward models evaluate intermediate steps. Because learned verifiers can be confidently wrong, production systems should validate calibration and downstream policy behavior.
Open-source verifiers make reward behavior inspectable, reproducible, and adaptable. Researchers can audit model weights and training methods, test failure modes, fine-tune for domain-specific constraints, and compare learned judgments with deterministic checks or human review. They do not guarantee alignment: reward hacking, distribution shift, bias, and verifier errors still require monitoring and layered evaluation.
A standard outcome reward model usually scores the completed response as a whole. A process reward model assigns judgments to intermediate reasoning steps, giving denser feedback and helping identify where a solution first goes wrong. PRMs can improve search and training credit assignment, but step-level labels are expensive and a locally plausible step does not always imply a correct final solution.
The 7B and 8B checkpoints can run on commodity systems with enough memory, especially after quantization, but GPUs provide substantially better throughput for RL training and batched verification. ThinkPRM-14B requires a larger memory budget. Exact requirements depend on precision, quantization, context length, batch size, and runtime; benchmark the intended configuration rather than estimating from parameter count alone.
Serve the verifier behind a batched inference function, send it policy-generated candidates or reasoning traces, normalize its judgments into rewards, and feed those rewards to an RL framework such as TRL or verl. Modal can host verifier and policy inference, distribute rollout and scoring jobs, run GPU training, and execute code-backed checks inside isolated Sandboxes. Start with offline evaluation and shadow scoring before allowing a new verifier to control policy updates.