ExploitGym AI benchmark source code
ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
Uh oh!
There was an error while loading. Please reload this page.
Notifications You must be signed in to change notification settings
Fork 76
Star 576
BranchesTags
Open more actions menu
Folders and files
NameName
Last commit message
Last commit date
Latest commit
History
28 Commits
28 Commits
agent_scorer
agent_scorer
data
data
docs
docs
examples
examples
scripts/setup
scripts/setup
src/cybergym
src/cybergym
tests
tests
.gitignore
.gitignore
.pre-commit-config.yaml
.pre-commit-config.yaml
CHANGELOG.md
CHANGELOG.md
DATA_LICENSE.md
DATA_LICENSE.md
LICENSE
LICENSE
README.md
README.md
pyproject.toml
pyproject.toml
pytest.ini
pytest.ini
uv.lock
uv.lock
Repository files navigation
ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.
Quick start
1. Python deps
uv sync --extra proxy
2. Build runtime artifacts (gdb, socat, nc, node + agent CLIs) and
extract task data
bash scripts/setup/setup_data.sh
3. Verify the install
bash scripts/setup/validate.sh
4. Pull the Firewall Squid image
docker pull ubuntu/squid:latest
5. Pull the Docker images for the tasks you want to run
uv run scripts/setup/pull_images.py data/task_ids/sample.txt
6. Start the controller, firewall, and LLM proxy. pre_run.py runs the
readiness checks and starts all three (auto-detecting any already
running), or start them by hand — see docs/eval.md
export OPENAI_API_KEY=... export ANTHROPIC_API_KEY=... uv run scripts/setup/pre_run.py data/task_ids/sample.txt
7. Run the agent
export CYBERGYM_ADMIN_KEY=... uv run examples/run_agent.py --help
Detailed setup steps (system dependencies, GDB, static node, agent CLIs) live in docs/setup.md.
Documentation
Setup: Python deps, GDB, socat/nc, node + agent CLIs
Docker images: pulling target images per task family
Evaluation: controller / firewall / LLM proxy + examples/run_agent.py
Defenses: disabling system defenses (ASLR, etc.)
Firewall: outbound network isolation for agent containers
Submission: submission format and requirements for the benchmark leaderboard
Benchmark updates
The released benchmark is actively maintained. The current release is v1.0 with 869 instances. See CHANGELOG.md for the full version history. The canonical task list for the current release is data/task_ids/v1.txt.
Citation
If you use ExploitGym in your research, please cite:
@article{wang2026exploitgym, title={ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?}, author={Wang, Zhun and Schiller, Nico and Li, Hongwei and Sesha Narayana, Srijiith and Nasr, Milad and Carlini, Nicholas and Qi, Xiangyu and Wallace, Eric and Bursztein, Elie and Invernizzi, Luca and Thomas, Kurt and Shoshitaishvili, Yan and Guo, Wenbo and He, Jingxuan and Holz, Thorsten and Song, Dawn}, journal={arXiv preprint arXiv:2605.11086}, year={2026} }
License
The source code is licensed under Apache-2.0. The bundled task data under data/tasks/ derives from external upstreams and retains their respective licenses, see DATA_LICENSE.md.
Apache-2.0 license
Activity
Custom properties
Stars
576 stars
Watchers
4 watching
Forks
76 forks
Report repository