Skip to content
AI News HubLIVE
Original source2 min read

SimHack: Predicting the onset of detectable reward hacking before RL training

Summary

The paper introduces SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and predicts, before an actual RL run, when detectable reward hacking becomes dominant under a target recipe, enabling lower-risk recipe selection.

SourcearXiv Machine LearningAuthor: Junwei Deng, Alex Bie, Natalia Ponomareva
SimHack: Predicting the onset of detectable reward hacking before RL training
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 20 Oct 2026]

Title:SimHack: Predicting the onset of detectable reward hacking before RL training

View a PDF of the paper titled SimHack: Predicting the onset of detectable reward hacking before RL training, by Junwei Deng and 2 other authors

View PDF HTML (experimental)

Abstract:When training LLMs to code with reinforcement learning, reward hacking is often detectable but difficult to prevent, since (a) environment hardening is increasingly difficult as training environments grow more complex; and (b) training-time interventions incentivize models to obfuscate hacking behavior to bypass detection.

We instead study whether we can predict when detectable reward hacking becomes dominant before executing an RL run, enabling us to select lower-risk recipes in advance. We find that the initial hacking rate alone does not determine onset, which varies substantially across training recipes and repeated runs. We thus formulate this problem as predicting the distribution of the reward hacking onset step under a target training recipe.

We introduce SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulation runs of unseen target recipes without costly retraining. Across unseen target recipes, SimHack predicts the median onset to a relative error of 11.3% and achieves across-recipe rank correlation of 0.85, while generalizing to differences in model scale, curriculum, number of rollouts, and learning rate between reference and target recipes. As an application, we show that SimHack can guide training decisions by screening candidate RL curricula, discovering curricula over the same set of scenarios that train for 96.6% longer without hacking.

Comments: 19 pages, 7 figures

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2610.28707 [cs.LG]

(or arXiv:2610.28707v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2610.28707

arXiv-issued DOI via DataCite

Submission history

From: Junwei Deng [view email] [v1] Tue, 20 Oct 2026 15:58:47 UTC (669 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled SimHack: Predicting the onset of detectable reward hacking before RL training, by Junwei Deng and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.LG

new | recent | 2026-10

Change to browse by:

cs

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender (What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Reframes reward hacking as predicting the distribution of onset step under a target recipe rather than intervening during training.
  • SimHack learns dynamics from a few small-scale reference RL runs and simulates unseen recipes without costly retraining.
  • Achieves 11.3% relative error on median onset and 0.85 across-recipe rank correlation for unseen target recipes.
  • Can screen curricula, finding ones that train 96.6% longer without hacking over the same scenarios.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Research