Scaling Agentic RL: 365,000 Environments for SWE, Terminal, and Search

Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Search

The open research ecosystem has produced many great datasets for the three main agentic domains - software engineering, terminal use, and web research - but every one of them ships with its own harness, its own image conventions, its own grading scripts, and its own failure modes.

23 tasksets behind one taskset API

DOMAIN · TASKSETSTASKS

SOFTWARE

ENGINEERING

swesmith 83,519openswe 36,884swerebench_v2 32,079scaleswe 17,202swelego 15,903multiswe 6,835r2e_gym 4,578swebench_pro 731swebench_verified 500swebench_multilingual 300senior_swe_bench 50

~012345678901012345678901234567890123456789012345678,012345678901234560123456789001234567890

TERMINAL

tmax 14,600terminal_lego ~13,800openthoughts_tblite 100terminal_bench_2 89

~01234567890120123456789012345678,012345678901234560123456789001234567890

SEARCH

papersearchqa 59,907wideseek 44,632s1_deepresearch ~15,000openseeker 11,677deepdive 3,250browsecomp 1,266redsearcher 1,000browsecomp_plus 830

~01234567890101234567890123012345678901234567,012345678901234560123456789001234567890

TOTAL23 tasksets~01234567890123012345678901234560123456789012345,012345678900123456789001234567890

We integrated them all. We ship first-party integrations for 23 agentic tasksets behind one taskset API, unified runtime/sandbox hooks, and one command: ~198,000 software engineering tasks across 20+ languages, ~28,600 terminal tasks, and ~137,600 search tasks - ~365,000 tasks in total, ready for evals and RL training on Prime Intellect infrastructure, with validated and cleaned dataset re-uploads where the originals needed fixing.

uv pip install "git+https://github.com/PrimeIntellect-ai/research-environments.git#subdirectory=environments/swe/scaleswe_v1"

Example running ScaleSWE in Codex harness on Prime Sandboxes:

uv run eval scaleswe-v1 --harness.id codex --harness.runtime.type prime -n 3

What makes that single command possible is verifiers v1, which decomposes an environment into three independent layers: a taskset, a harness, and a runtime. This post is the taskset piece of that story.

One contract, three domains

Each upstream taskset made reasonable choices for its own harness - and those choices don't compose. SWE-bench applies test patches inside a generated eval script; R2E-Gym bakes tests into the image and compares against expected outputs; search benchmarks each invent their own judge. If you want to train one agent across all of them, you need those lifecycles normalized without breaking each taskset's own scoring semantics.

Our integrations keep every taskset's original grading path - upstream log parsers, upstream report generation, upstream test commands - and normalize everything around it:

One API. Every taskset loads from a typed config (dataset, split, filters), provisions a sandbox from the task's image, and scores with the taskset's own logic. Adding filter_fn or swapping splits works the same everywhere.

One image registry. Task images live in our own Prime image registry, co-located with the sandboxes - ~135,000 prebuilt open-source task images, to our knowledge the largest such catalog hosted by any sandbox provider. No flags, no namespace mapping, and no Docker Hub rate limits at rollout time - the ones you would otherwise hit immediately when running a thousand concurrent rollouts against Docker Hub. You can also pre-build and upload your own images (prime images push) or copy existing ones straight from Docker Hub (prime images transfer-bulk).

One integrity standard. During a rollout the agent lives inside the same sandbox as the grading machinery, so anything readable in the container is fair game for a reward hack. Every integration therefore withholds grading material - test patches, expected outputs, grading scripts - until scoring time. Some original authors chose to keep it visible: R2E-Gym ships its grading tests readable inside the image at /r2e_tests, and Multi-SWE's containers carry the grading shell scripts and test.patch readable under /home. That is a fine choice for their attach-at-eval harnesses, but not for a live RL sandbox, so our integrations hide these artifacts and restore them only for scoring.

One validation bar. Before a dataset earns a spot as default, we run gold-patch and no-op validation against it and re-upload cleaned versions with the exclusions preserved for transparency - the next section covers how.

A common shape for a taskset is a data schema plus a handful of hooks:

import verifiers.v1 as vf

class MyTaskData(vf.TaskData): base_commit: str # state the sandbox resets to test_patch: str # grading material - withheld until scoring gold_patch: str # reference fix - used only by validate

class MyTask(vf.Task[MyTaskData]): async def setup(self, runtime: vf.Runtime): ... # prepare the repo inside the task's image

async def finalize(self, trace: vf.Trace, runtime: vf.Runtime): ... # capture the agent's diff into the trace for posterity

@vf.reward async def solved(self, runtime: vf.Runtime) -> float: ... # restore tests, apply test_patch, run the upstream grading path

async def apply_gold_patch(self, runtime: vf.Runtime): ... # apply the reference fix

async def validate(self, runtime: vf.Runtime) -> bool: ... # gold patch must score 1.0; not calling validate must not

class MyTaskset(vf.Taskset[MyTask, vf.TasksetConfig]): def load(self) -> list[MyTask]: ... # (dataset, split, filter_fn) -> tasks pinned to images

We constantly evaluate or train on these tasksets internally, which continuously validates their interplay with our sandbox infrastructure, and we update the datasets, images, and integrations on a regular basis.

Validated re-uploads

A taskset row is only useful if it can produce a clean reward signal: gold patch applied → tests pass; no patch → tests fail. A surprising fraction of open agentic task data fails this precondition - broken images, network-dependent tests, drifted expected outputs, and tasks solvable without touching the code at all.

So we validated. For each dataset we ran the gold patch through the full scoring path in fresh sandboxes, retried failures up to 10× to separate flaky from deterministically broken, ran independent second passes to catch noisy rows, and ran multiple no-edit passes to drop tasks that score 1.0 without any fix. The results are re-uploaded in the SWE RL collection with the dropped rows persisted, so you can audit every exclusion:

R2E-Gym-Subset-Verified: 4,522 of 4,578 rows pass 10×-retry gold validation; the 56 drops are mostly network/timing-sensitive aiohttp/tornado tests.

SWE-Lego-Real-Data-Verified: 4,323 of 4,432 rows, same methodology.

Multi-SWE-RL-Verified: 2,232 rows surviving two gold-patch passes plus a no-edit filter that caught tasks gradeable as solved with zero edits.

SWE-rebench-V2-Filtered-Verified: 6,275 rows from 32,079 raw - language-level drops for wholesale-broken images, two independent gold passes with flaky rows removed, and problem statements scrubbed of inline GitHub issue/PR references.

SWE-Bench-Verified-Quick: 468 of 500 Verified instances, dropping the slowest examples for quick online-evals.

The tooling behind these passes ships with verifiers, so you can reproduce them yourself. uv run validate is the model-free sibling of eval: for every task it runs the gold check (setup, apply the reference fix, tests must pass) and a setup-only no-op check (tests must not pass without a fix) in independent runtimes - --only-gold and --only-setup select one side. And uv run debug --command "" (or --script-path) sets up a task's sandbox, runs an arbitrary command or uploaded script inside it, and persists the traces - by far the most convenient way to poke at a container task's state without writing a harness.

Software Engineering Tasks

SWE-bench Verified (Jimenez et al., Princeton - paper) is the canonical benchmark: real GitHub issues from major Python repos, resolved by writing a patch that makes hidden failing tests pass, human-filtered by OpenAI down to 500 instances confirmed to be fairly solvable. swebench_verified_v1 runs the Harbor Hub packaging against the official instance images.

SWE-bench Multilingual (Khandpur et al., SWE-bench team - blog, paper, swebench_multilingual_v1) extends the canonical benchmark beyond Python: the official 300-instance test set across C, C++, Go, Java, JS/TS, PHP, Ruby, and Rust. Our integration wraps the Harbor packaging, and grades with the canonical verifier.

SWE-bench Pro (Deng et al., Scale AI - repo) is the harder successor: 731 public tasks from license-friendly repos with large-scale diffs. Our integration (swebench_pro_v1) wraps the Harbor packaging and resolves each task's hosted image from its test config.

SWE-smith (Yang et al., Stanford / Princeton - paper, swesmith_v1) inverts the data problem: instead of mining issues, it injects bugs into healthy repos and keeps the tests that catch them - 88,130 instances across eight languages. Task branches carry the bug with the failing tests removed from the head commit; our port keeps the agent at that head, restores the tests via HEAD~1 only inside scoring, and reuses upstream's grading and profile registry verbatim.

R2E-Gym (Jain et al., UC Berkeley - paper) generates executable environments from real commits with synthesized issue descriptions - 4,578 instances in its curated subset. The author images keep the grading tests readable inside the container; r2e_gym_v1 restores /r2e_tests only for scoring. We publish a gold-validated re-upload - 4,522 of the 4,578 instances - which the taskset defaults to.

Multi-SWE (Zan et al., ByteDance - paper, multiswe_v1) extends beyond Python: 4,703 containerized RL instances across C, C++, Go, Java, JS, Rust, and TypeScript, graded by the upstream report protocol. Our re-uploads - the RL set plus the 2,132-task Multi-SWE-bench eval set - fix the HF schema, and we hide the grading scripts and test.patch that the original containers leave readable.

SWE-rebench-V2 (Badertdinov et al., Nebius - paper, swerebench_v2_v1) continuously mines fresh GitHub PRs into tasks - 32,079 of them across 20 languages, naturally decontaminated by recency. We vendor the upstream log parsers (~3.6k lines, one per language ecosystem) for grading fidelity, hide the test patch until scoring, and default to our filtered-and-verified re-upload.

Scale-SWE (Zhao et al. - paper, scaleswe_v1) contributes 20,181 Python tasks with test patches and scripts applied just before evaluation, exactly as the authors specify. Our re-upload carries the 17,202 instances that survived validation.

SWE-Lego (Tao et al., Huawei - paper) builds SWE-bench-style training data at scale; we host the 4,432 resolved real-data instances plus a gold-validated variant. swelego_v1 keeps the original authors' evaluation pattern: the failing tests stay hidden during the rollout and are applied only at scoring, after resetting any test files the agent touched back to base_commit.

OpenSWE (Fu et al., GAIR - paper) pairs 45,320 tasks with per-task evaluation scripts stored as data. openswe_v1 keeps the eval script out of the sandbox until scoring writes and runs it.

Senior SWE-Bench (Snorkel AI - repo, senior_swe_bench_v1) moves past issue-resolution: 50 public investigation and design tasks drawn from 12 production open-source repos (gitea, posthog, teleport, immich, and more), scored by the upstream verifier pipeline - native pytest/vitest checks plus optional LLM rubric judges. Every task runs from a prebuilt Prime image and is gold-validated against the upstream solutions.

All our filtered and verified SWE re-uploads live in the SWE RL collection on Hugging Face, each with a dataset card documenting exactly what changed vs upstream, the validation methodology, and exclusions preserved for audit.

Terminal Tasks

Terminal-Lego (Yang et al., Huawei / HKU - paper, terminal_lego_v1) is ~13,800 Docker-verif

[truncated for AI cost control]