Should we chaos test our agents?

Notifications You must be signed in to change notification settings

Fork 4

Star 24

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

23 Commits

23 Commits

.github/workflows

.github/workflows

apps

apps

assets

assets

evalmonkey

evalmonkey

scripts

scripts

tests

tests

ui

ui

.env.example

.env.example

.gitignore

.gitignore

LICENSE

LICENSE

README.md

README.md

demo_rag_app.sh

demo_rag_app.sh

pyproject.toml

pyproject.toml

schema.yml

schema.yml

Repository files navigation

Agent Benchmarking & Chaos Engineering Framework

"Don't just trust your agent. Prove it works. Then break it."

Overview

Agents are fundamentally non-deterministic. They rely on external APIs, tool loops, and massive context windows. EvalMonkey is the ultimate, strictly local, open-source execution harness that enables developers to:

🎯 Benchmark Capabilities: Run standard Agent benchmark datasets against your agent endpoints natively!

🔥 Inject Chaos: Mutate headers, spike latency, and corrupt schemas dynamically to prove true resilience.

📈 Track Production Reliability: Locally store all scores to visualize a single Production Reliability metric over time!

🛠 Generate Improvement Evals: When scores are poor, automatically synthesise targeted test cases using your LLM — then hand them to Claude Code or Cursor to fix your agent.

EvalMonkey natively supports evaluating ANY LLM: AWS Bedrock, Azure, GCP, OpenAI, and Ollama.

Note on API Keys: If you have special setups that generate long-lived, static API keys for Bedrock, Azure, or GCP, simply supply them in the .env! EvalMonkey seamlessly supports both standard IAM / Service Account credential flows and long-term stateless authentication strings.

🚀 At a Glance

11 Agent Frameworks natively supported: CrewAI, LangChain, LlamaIndex, LangGraph, Pydantic AI, OpenAI Agents, Microsoft AutoGen, AWS Bedrock, Ollama, Strands, and custom HTTP endpoints.

19 Standard Benchmarks out-of-the-box: GSM8K, BIG-Bench Hard, HotpotQA, ToxiGen, MT-Bench, MBPP, and more — all categorised by the agent type they target.

23 Chaos Injections ready to run: 12 client-side payload mutations + 11 server-side middleware injections — all text-based, no GPU or vision dependencies.

Automatic Eval Asset Generation: Poor benchmark scores automatically produce traces.json, evals.json, and improvement_prompt.md — one cat command away from Claude Code or Cursor.

📊 EvalMonkey Web Dashboard

Visualize all your benchmark runs, track reliability scores over time, and inspect failure traces interactively!

EvalMonkey Main Dashboard showing scenario trends and score histories.

Deep-dive into specific benchmark runs and chaos tests.

⚡️ Quick Start

Option A — Let Claude Code or Cursor set it up for you (30 seconds)

Open Claude Code, Cursor, or any AI coding assistant and paste this prompt:

Set up EvalMonkey in my project so I can benchmark my AI agent.

1. Clone https://github.com/Corbell-AI/evalmonkey into a sibling folder
2. Run: pip install -e . inside that folder
3. Copy .env.example to .env and ask me which LLM provider I want to use as the benchmark judge (OpenAI, Anthropic, Bedrock, or Ollama) — then fill in the correct key
4. Run: evalmonkey init --framework --name "My Agent" --port

Use the framework my agent is built with (crewai / langchain / openai / bedrock / autogen / ollama / strands / custom)

1. Show me the generated evalmonkey.yaml and ask me to confirm the agent URL and response path are correct
2. Run a quick smoke test: evalmonkey run-benchmark --scenario gsm8k --sample-agent rag_app --limit 2

to confirm everything is wired up correctly

1. Then run the real benchmark against my agent: evalmonkey run-benchmark --scenario mmlu --limit 5
2. Show me the score and explain what it means

The agent will handle cloning, installing, configuring your .env, and running the first benchmark — all without you typing a single command.

Option B — Manual Setup (5 minutes)

1. Install

git clone https://github.com/Corbell-AI/evalmonkey cd evalmonkey pip install -e .

1. Configure your LLM key (used only as the evaluation judge — never for your agent)

cp .env.example .env

Open .env and set one of these depending on your LLM provider:

EVAL_MODEL=gpt-4o OPENAI_API_KEY=sk-... # OpenAI

— OR —

EVAL_MODEL=anthropic/claude-haiku-4-5 ANTHROPIC_API_KEY=sk-ant-... # Anthropic

— OR —

EVAL_MODEL=bedrock/anthropic.claude-3-haiku-20240307-v1:0 AWS_ACCESS_KEY_ID=... # AWS Bedrock

— OR — (no key needed)

EVAL_MODEL=ollama/llama3 # Local Ollama

1. Smoke test with the built-in sample agent (no agent of your own needed yet)

evalmonkey run-benchmark --scenario gsm8k --sample-agent rag_app --limit 3

You should see 3 samples run and a score printed. ✅

1. Point it at your own agent

cd /path/to/your/agent/project evalmonkey init --framework crewai --name "My Agent" --port 8000

Edit the generated evalmonkey.yaml to set your agent's URL and response format

evalmonkey run-benchmark --scenario mmlu --limit 5

evalmonkey.yaml is discovered from the current working directory — same convention as pytest and docker-compose.

🤝 Works With Any Agent — No Code Changes Required

EvalMonkey talks to your agent over plain HTTP. As long as your agent is running and has an endpoint URL, you're done. That's it.

Point EvalMonkey at your existing running agent

evalmonkey run-benchmark --scenario mmlu --target-url http://localhost:8000/chat

Your agent returns a different JSON format? Use two flags to map any request/response shape:

Flag What it does Example

--request-key Which key to send the question under message, prompt, input

--response-path Dot-path to extract the answer from output.text, choices.0.message.content, result

CrewAI agent that takes {"message":""} and returns {"reply":""}

evalmonkey run-benchmark --scenario mmlu \ --target-url http://localhost:8000/chat \ --request-key message \ --response-path reply

OpenAI-compatible endpoint returning {"choices":[{"message":{"content":""}}]}

evalmonkey run-benchmark --scenario arc \ --target-url http://localhost:8000/v1/chat/completions \ --request-key content \ --response-path choices.0.message.content

Supported Frameworks

Framework Notes

🦜 LangChain Any Chain, LCEL pipe, or AgentExecutor behind FastAPI

🦙 LlamaIndex Any QueryEngine, ChatEngine, or ReActAgent

🕸️ LangGraph Any compiled StateGraph or MessageGraph

🛡️ Pydantic AI Any validated Agent returning structured or text data

🤖 CrewAI Any Crew behind a /chat or custom endpoint

✨ OpenAI Agents SDK Native OpenAI Chat Completions format supported via --response-path

☁️ AWS Bedrock / Agent Core Any Bedrock endpoint, IAM or long-lived key

🧩 Microsoft AutoGen Any ConversableAgent behind HTTP

🦙 Ollama Running locally at http://localhost:11434

🧬 Strands Enterprise support agents and chatbots

🌐 Any HTTP Agent Flask, Express.js, Go — if it accepts POST it works

📦 Don't have an HTTP endpoint yet? Use our ready-made thin adapters (click to expand)

Copy the relevant file from apps/framework_adapters/ next to your agent code, swap in your Crew/Chain/Agent, and run it. No changes needed to EvalMonkey.

langchain_adapter.py — wraps any LangChain chain

crewai_adapter.py — wraps any CrewAI Crew

openai_agents_adapter.py — wraps OpenAI Agents SDK

bedrock_agentcore_adapter.py — wraps AWS Bedrock Converse API

autogen_adapter.py — wraps Microsoft AutoGen Crew

Each adapter is ~40 lines and exposes a /solve endpoint on localhost.

🌍 Supported Standard Benchmarks

EvalMonkey natively supports 19 off-the-shelf benchmark datasets pulled directly from HuggingFace. All benchmarks are text-only — no vision, audio, or multimodal agent required. List them anytime via the CLI:

evalmonkey list-benchmarks

Scenario ID Agent Category Description

gsm8k 🧠 Reasoning Grade School Math word problems — multi-step arithmetic & logic.

xlam 🔧 Tool Use XLAM Function Calling 60k — tool execution & parameter structuring.

swe-bench 💻 Coding SWE-Bench — resolve real-world GitHub issues from a description only.

gaia-benchmark 🔍 Research GAIA — multi-step real-world tasks requiring web/tool chaining.

human-eval 💻 Coding HumanEval — Python function synthesis from docstrings.

mmlu 💬 Q&A MMLU — general knowledge across 57 academic subjects.

arc 🧠 Reasoning ARC Challenge — hard grade-school science multiple-choice.

truthfulqa 🛡️ Safety TruthfulQA — detects hallucination and human-like falsehood mimicry.

hella-swag 🧠 Reasoning HellaSwag — commonsense sentence-completion inference.

bbh 🧠 Reasoning BIG-Bench Hard — 23 tasks where LLMs still fall below human baselines.

winogrande 💬 Q&A WinoGrande — pronoun disambiguation resistant to dataset shortcuts.

drop 🔍 Research DROP — reading comprehension with embedded numerical & date math.

natural-questions 💬 Q&A Natural Questions — real Google search queries with Wikipedia answers.

hotpotqa 🔍 Research HotpotQA — multi-hop reasoning across two Wikipedia documents.

mbpp 💻 Coding MBPP — entry-level Python function synthesis from plain English.

apps 💻 Coding APPS — competitive-programming & interview-style code challenges.

mt-bench 📋 Instruction Following MT-Bench — multi-turn dialogues across writing, roleplay, reasoning, STEM.

alpacaeval 📋 Instruction Following AlpacaEval — instruction quality judged by GPT-4 head-to-head.

toxigen 🛡️ Safety ToxiGen — detects toxic/hateful content generation across 13 demographic groups.

🛠️ Build Your Own Custom Benchmarks (click to expand)

Yes, people absolutely bring their own datasets! The most powerful way to test an agent is to grab 10-50 real questions from your production logs, dump them into a CSV, and evaluate your agent against them.

EvalMonkey natively supports auto-parsing .yaml, .json, and .csv files!

You don't need any complex ETL pipelines. Just drop a file (e.g. evals.csv, evals.json, or custom_evals.yaml) in your execution directory and pass it to EvalMonkey!

1. CSV Example (evals.csv)

If using a CSV, just make sure you have the columns id and expected_behavior_rubric. Any other column you add (like question, topic, image_url) will be automatically gathered and sent in the JSON payload directly to your agent!

id expected_behavior_rubric question

get_benefits Must return the URL linking to the company hr portal Where do I sign up for medical benefits?

time_off Provide the exact number of standard vacation days (15) How many days of PTO do I get?

evalmonkey run-benchmark --scenario get_benefits --eval-file evals.csv

1. JSON / YAML Example (evals.json)

If you use JSON or YAML, you must nest the agent payload keys explicitly under an input_payload dict object:

[ { "id": "onboarding_query", "description": "Test HR agent's ability to return the onboarding link.", "expected_behavior_rubric": "Must contain exactly the URL https://hr.example.com/benefits", "input_payload": { "question": "Where do I sign up for benefits?" } } ]

evalmonkey run-benchmark --scenario onboarding_query --eval-file evals.json

🛠️ Experiences

Experience 1: Local Sample Agents (Single Command Start)

Easiest Experience: Test our built-in sample agents with a single command! EvalMonkey will spawn the sample agent in the background automatically and run the benchmark.

Run against just the first 5 records

evalmonkey run-benchmark --scenario gsm8k --sample-agent rag_app

Run a statistically robust test against 50 different records!

evalmonkey run-benchmark --scenario gsm8k --sample-agent rag_app --limit 50

Metrics Output:

╭──────────────────────────────────────────────────────────╮ │ Benchmark Results │ │ ──────────────────────────────────────────────────────── │ │ Scenario gsm8k │ │ Score 90/100 (Diff: +5) │ │ Previous 85/100 │ │ Reasoning Agent correctly utilized calculator for ..

[truncated for AI cost control]