TLDR: AI environments are becoming a core ingredient of model capability: executable worlds that generate trajectories and feedback, not just exams. This essay traces their path from Atari, OpenAI Gym, and AlphaZero through MuJoCo to the RLVR-driven post-training behind Tulu 3 and DeepSeek-R1, then surveys the new generation of agent environments, including SWE-Gym, WebArena, OSWorld, TheAgentCompany, Tau2-bench, and AgentMercury.
It then lays out what makes a good environment: rewards that track the real outcome and resist hacking, causal fidelity, affordable and repeatable practice, structured difficulty, and improvement that transfers outside the training world. The frontier, the essay argues, is the supply of varied, consequential practice with dependable feedback.
Imagine hiring an engineer who has read every programming book but has never run a program. They can explain databases, discuss distributed systems, and produce an impressively confident architecture diagram. Then you ask them to fix a failing deployment. Suddenly, the difference between describing work and performing it becomes rather expensive.
This is a useful way to think about AI environments. Pretraining gives a model broad knowledge from recorded experience. An environment lets an agent act, encounter consequences, and generate new experience. A compiler rejects its code. A browser reveals an unexpected page. A customer changes the request halfway through the workflow.
My thesis is that environments are becoming a major ingredient in AI capability. They are generators of training data: executable worlds that produce trajectories and feedback. Their quality determines what an agent can practice, which mistakes it can discover, and whether improvement inside the training setup means anything outside it.
From game simulators to language models
Read more