Uh oh!
There was an error while loading. Please reload this page.
Notifications You must be signed in to change notification settings
Fork 0
Star 0
BranchesTags
Open more actions menu
Folders and files
NameName
Last commit message
Last commit date
Latest commit
History
2 Commits
2 Commits
.agentspec
.agentspec
.github/workflows
.github/workflows
examples
examples
landing
landing
src
src
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
SHOW-HN.md
SHOW-HN.md
action.yml
action.yml
package-lock.json
package-lock.json
package.json
package.json
tsconfig.json
tsconfig.json
Repository files navigation
Testing framework for AI agents — Jest for non-deterministic AI behavior
AI agents are non-deterministic. When you change a prompt, swap a model, or update a tool, behavior shifts in ways you can't predict. AgentSpec lets you write tests that catch those shifts before they reach production.
Quick start
npm install -g agentspec agentspec init
This creates agentspec.yaml:
name: "my-agent-tests" tests:
- name: "handles greeting"
input: "hello" expect: contains: "help"
Run tests:
agentspec run
Assertions for non-deterministic output
AgentSpec provides assertions designed for AI output, not just string equality:
tests:
- name: "handles expired token"
input: "my token expired" expect: contains: "refresh token" not_contains: "I don't know" max_latency_ms: 5000
- name: "routes billing question"
input: "I want a refund" expect: contains_any: ["billing", "refund", "support"] tool_called: "transfer_to_billing"
- name: "response is semantically on-topic"
input: "how do I reset my password?" expect: semantically_similar: "reset password credentials" min_confidence: 0.5
- name: "JSON output has correct structure"
input: "get user profile" expect: json_path: "data.email" json_value: "[email protected]"
- name: "matches error code pattern"
input: "simulate error" expect: regex: "ERR-\\d{5}"
LLM-as-judge
Use a local model to evaluate response quality — no API costs, full privacy:
Use local Ollama as judge
agentspec run --judge-endpoint http://127.0.0.1:11434/v1/chat/completions --judge-model qwen2.5:7b
tests:
- name: "response is helpful"
input: "How do I reset my password?" expect: llm_judge: "The response should explain how to reset a password"
Behavior diff reports
AgentSpec stores the last passing output per test. When behavior changes, you see exactly what shifted:
⚠️ REGRESSION support agent > handles expired token ⚠ Behavior REGRESSED — test was passing, now failing + added: your, token, has, expired, please, contact, support
- removed: you, need, refresh, the, token, settings, security
Testing real agents
Use --endpoint to test any HTTP-accessible agent:
agentspec run --endpoint https://my-agent.example.com/chat
AgentSpec POSTs {input: "..."} to your endpoint and expects {output: "..."} in the response.
CI/CD integration
Exit code 1 on failure
agentspec run --ci
JUnit XML for CI systems
agentspec run --junit > results.xml
JSON output
agentspec run --json
GitHub Action
- uses: agentspec/action@v1
with: test-dir: tests format: junit
CLI commands
agentspec run [--dir tests] [--ci] [--json] [--junit] [--watch] [--test "pattern"] agentspec init agentspec list [--dir tests] agentspec version
Why AgentSpec?
Problem AgentSpec
Agent behavior changes with model updates Regression tests catch the shift
"It worked yesterday" debugging Diff reports show what changed
Manual testing before each deploy CI/CD gates with --ci exit codes
Jest/Vitest don't handle non-deterministic output contains_any, regex, llm_judge, semantically_similar
No CI integration for AI features GitHub Action + JUnit XML
Roadmap
YAML test definitions
Core assertions (contains, not_contains, contains_any, contains_all, regex)
Metadata assertions (max_latency_ms, max_tokens, tool_called)
Semantic similarity (word overlap)
JSON path assertions
CLI with run/init/list commands
CI mode with exit codes
JUnit XML and JSON output
Test filtering
GitHub Action
Behavior diff reports — see what changed between runs
LLM-as-judge — use a local model (Ollama) to evaluate output quality
HTTP agent adapter — test real agents via --endpoint
Auto-test generation from conversation history
Python agent support
Watch mode with file watcher
Web dashboard
GitHub PR integration (comment on PRs with behavior diffs)
License
MIT
About
Testing framework for AI agents — Jest for non-deterministic AI behavior
Resources
Readme
License
MIT license
Uh oh!
There was an error while loading. Please reload this page.
Activity
Custom properties
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
No releases published
Packages 0
Uh oh!
There was an error while loading. Please reload this page.
Contributors
Uh oh!
There was an error while loading. Please reload this page.
Languages
TypeScript 66.2%
HTML 22.6%
JavaScript 11.2%