Answerwatch – Track changes in what AI models recommend

Notifications You must be signed in to change notification settings

Fork 0

Star 0

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

4 Commits

4 Commits

.github

.github

src/answerwatch

src/answerwatch

tests

tests

.gitignore

.gitignore

CHANGELOG.md

CHANGELOG.md

CODE_OF_CONDUCT.md

CODE_OF_CONDUCT.md

CONTRIBUTING.md

CONTRIBUTING.md

LICENSE

LICENSE

README.md

README.md

ROADMAP.md

ROADMAP.md

SECURITY.md

SECURITY.md

SUPPORT.md

SUPPORT.md

pyproject.toml

pyproject.toml

setup.py

setup.py

Repository files navigation

Track what AI models recommend—and see when the answer changes.

Your brand, product, or category can look different depending on which model someone asks. Answerwatch makes that visible: run the same prompt across models, compare their recommendations, and keep a local audit trail as answers drift.

Everything stays local in SQLite. The only external call is the model request you choose to make.

┌─ OpenAI prompts ───────────├─ Anthropic ──► ranked entities ──► agreement + drift └─ Google │ ▼ static HTML report

Start here — two-minute demo

Requires Python 3.10 or newer.

python -m pip install -e . answerwatch init answerwatch run --demo answerwatch compare answerwatch report

Open the HTML path printed by the last command. The demo makes no network requests and spends no API credits. It produces a polished dashboard with agreement, consensus, rank gaps, citation domains, and raw answers.

Why Answerwatch?

You need to know Answerwatch shows

Do models agree? Pairwise recommendation-agreement matrix

What does only one model mention? Unique recommendations per model

Is ranking materially different? Largest rank gaps for shared entities

Did the answer change this week? Run-to-run drift with additions and removals

Can I trust the analysis? Raw-response audit trail and extraction evidence

Run with live models

Answerwatch uses OpenRouter so one API key can query models from different providers.

export OPENROUTER_API_KEY="..." answerwatch run answerwatch compare answerwatch report

answerwatch init creates:

answerwatch.yml

models:

• openai/gpt-5.2
• anthropic/claude-sonnet-4.5
• google/gemini-3-flash-preview

prompts: sample-prompts.yml database: .answerwatch/answerwatch.db

settings: temperature: 0.2 max_tokens: 1200 retries: 2

sample-prompts.yml

prompts:

• id: best-running-shoes

text: What are the best running shoes for beginners? entity: product

Use model identifiers currently available in OpenRouter.

Commands

answerwatch init answerwatch run [--demo] answerwatch compare [RUN_ID] [--baseline RUN_ID|previous] answerwatch report [RUN_ID] [--baseline RUN_ID|previous] [--output report.html]

Without a run ID, compare and report use the latest completed run.

To see recommendation drift, run once, change time or model settings, then:

answerwatch compare --baseline previous answerwatch report --baseline previous

How extraction works

Version 0.1 asks models to put named recommendations first in a numbered list. A deterministic local parser extracts those list items. Both the raw answer and the exact extracted evidence are retained in SQLite.

This keeps the first release fast, private, and auditable. It does not claim that generated URLs are verified citations.

Local data

The default database and reports live under .answerwatch/. Delete that directory to remove local run history.

Development

PYTHONPATH=src python -m unittest discover -s tests -v

Please read CONTRIBUTING.md before opening a pull request. We use Conventional Commit-style messages, for example:

feat(report): surface consensus recommendations fix(extract): preserve linked entity names docs: clarify live model setup

Roadmap

Now — local runs, recommendation comparison, static reports, and OpenRouter support.

Next — stronger entity normalization, structured provider citations, and scheduled local runs.

Later — optional hosted history, alerts, teams, and branded reports.

See ROADMAP.md for the intended boundaries.

Community and security

Contributing

Code of Conduct

Security policy

Support

License

MIT

MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy

Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks

Report repository