AgentSpec: Testing framework for AI agents (Jest for non-deterministic behavior)

Uh oh!

There was an error while loading. Please reload this page.

Notifications You must be signed in to change notification settings

Fork 0

Star 0

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

2 Commits

2 Commits

.agentspec

.agentspec

.github/workflows

.github/workflows

examples

examples

landing

landing

src

src

tests

tests

.gitignore

.gitignore

LICENSE

LICENSE

README.md

README.md

SHOW-HN.md

SHOW-HN.md

action.yml

action.yml

package-lock.json

package-lock.json

package.json

package.json

tsconfig.json

tsconfig.json

Repository files navigation

Testing framework for AI agents — Jest for non-deterministic AI behavior

AI agents are non-deterministic. When you change a prompt, swap a model, or update a tool, behavior shifts in ways you can't predict. AgentSpec lets you write tests that catch those shifts before they reach production.

Quick start

npm install -g agentspec agentspec init

This creates agentspec.yaml:

name: "my-agent-tests" tests:

• name: "handles greeting"

input: "hello" expect: contains: "help"

Run tests:

agentspec run

Assertions for non-deterministic output

AgentSpec provides assertions designed for AI output, not just string equality:

tests:

• name: "handles expired token"

input: "my token expired" expect: contains: "refresh token" not_contains: "I don't know" max_latency_ms: 5000

• name: "routes billing question"

input: "I want a refund" expect: contains_any: ["billing", "refund", "support"] tool_called: "transfer_to_billing"

• name: "response is semantically on-topic"

input: "how do I reset my password?" expect: semantically_similar: "reset password credentials" min_confidence: 0.5

• name: "JSON output has correct structure"

input: "get user profile" expect: json_path: "data.email" json_value: "[email protected]"

• name: "matches error code pattern"

input: "simulate error" expect: regex: "ERR-\\d{5}"

LLM-as-judge

Use a local model to evaluate response quality — no API costs, full privacy:

Use local Ollama as judge

agentspec run --judge-endpoint http://127.0.0.1:11434/v1/chat/completions --judge-model qwen2.5:7b

tests:

• name: "response is helpful"

input: "How do I reset my password?" expect: llm_judge: "The response should explain how to reset a password"

Behavior diff reports

AgentSpec stores the last passing output per test. When behavior changes, you see exactly what shifted:

⚠️ REGRESSION support agent > handles expired token ⚠ Behavior REGRESSED — test was passing, now failing + added: your, token, has, expired, please, contact, support

• removed: you, need, refresh, the, token, settings, security

Testing real agents

Use --endpoint to test any HTTP-accessible agent:

agentspec run --endpoint https://my-agent.example.com/chat

AgentSpec POSTs {input: "..."} to your endpoint and expects {output: "..."} in the response.

CI/CD integration

Exit code 1 on failure

agentspec run --ci

JUnit XML for CI systems

agentspec run --junit > results.xml

JSON output

agentspec run --json

GitHub Action

• uses: agentspec/action@v1

with: test-dir: tests format: junit

CLI commands

agentspec run [--dir tests] [--ci] [--json] [--junit] [--watch] [--test "pattern"] agentspec init agentspec list [--dir tests] agentspec version

Why AgentSpec?

Problem AgentSpec

Agent behavior changes with model updates Regression tests catch the shift

"It worked yesterday" debugging Diff reports show what changed

Manual testing before each deploy CI/CD gates with --ci exit codes

Jest/Vitest don't handle non-deterministic output contains_any, regex, llm_judge, semantically_similar

No CI integration for AI features GitHub Action + JUnit XML

Roadmap

YAML test definitions

Core assertions (contains, not_contains, contains_any, contains_all, regex)

Metadata assertions (max_latency_ms, max_tokens, tool_called)

Semantic similarity (word overlap)

JSON path assertions

CLI with run/init/list commands

CI mode with exit codes

JUnit XML and JSON output

Test filtering

GitHub Action

Behavior diff reports — see what changed between runs

LLM-as-judge — use a local model (Ollama) to evaluate output quality

HTTP agent adapter — test real agents via --endpoint

Auto-test generation from conversation history

Python agent support

Watch mode with file watcher

Web dashboard

GitHub PR integration (comment on PRs with behavior diffs)

License

MIT

About

Testing framework for AI agents — Jest for non-deterministic AI behavior

Resources

Readme

License

MIT license

Uh oh!

There was an error while loading. Please reload this page.

Activity

Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks

Report repository

Releases

No releases published

Packages 0

Uh oh!

There was an error while loading. Please reload this page.

Contributors

Uh oh!

There was an error while loading. Please reload this page.

Languages

TypeScript 66.2%

HTML 22.6%

JavaScript 11.2%