Show HN: AI PR reviewer built to stay quiet – 89% of merged PRs got no comments

Notifications You must be signed in to change notification settings Fork 0 Star 0 Add this Action to an existing workflow or create a new one View on Marketplace BranchesTags Open more actions menu Latest commit History 12 Commits 12 Commits Folders and files NameName Last commit message Last commit date .github/workflows .github/workflows bench-results bench-results dist dist docs docs scripts scripts src src test test .gitignore .gitignore LICENSE LICENSE README.md README.md SECURITY.md SECURITY.md action.yml action.yml package-lock.json package-lock.json package.json package.json tsconfig.json tsconfig.json tsup.config.ts tsup.config.ts Repository files navigation An AI PR reviewer built to eliminate review noise — not add to it. Most AI reviewers re-review the whole PR on every push and repeat themselves until the team mutes them. pr-sage is designed around the opposite goal: say each thing once, follow your team's rules, and stay silent when there is nothing new to say. 🔇 Zero duplicate comments. Findings carry content fingerprints — a line shift won't make the same comment appear twice, and re-runs post nothing when nothing changed. ✅ Finding lifecycle. Follow-up reviews report which findings remain unresolved and which were fixed. ⏩ Incremental by default. After the first review, only the commits you pushed since get reviewed. Less noise, fewer tokens. 📏 Your rules, not generic advice. .pr-sage.json instructions plus automatic CLAUDE.md/CONTRIBUTING.md injection make reviews follow team conventions. 🚦 A quality gate, not just commentary. --fail-on critical blocks merges; --event auto approves clean PRs and requests changes on real problems. 🖥️ Reviews before the PR exists. pr-sage local reviews your git diff pre-push — no server, no PR, no GitHub token. 🔐 Your keys, your data path. No server, nothing stored; code goes only to the provider you choose — Claude, OpenAI, or Gemini — or never leaves your machine at all with a self-hosted OpenAI-compatible endpoint (Ollama, vLLM, LM Studio). See SECURITY.md. Ships as a CLI, a GitHub Action, and a TypeScript library. Quick start The fastest path — an interactive wizard that writes your config and the GitHub Action workflow, and tells you exactly which secret to register: npx pr-sage init npx pr-sage doctor Or by hand (CLI): export GITHUB_TOKEN=ghp_... export ANTHROPIC_API_KEY=sk-ant-... npx pr-sage review --repo owner/name --pr 123 Preview without posting anything: npx pr-sage review --repo owner/name --pr 123 --dry-run Review your local changes before pushing (no PR, no GitHub token needed): npx pr-sage local --base main # diff vs main npx pr-sage local --staged --fail-on critical # gate staged changes Review in Korean with a different provider: export OPENAI_API_KEY=sk-... npx pr-sage review --repo owner/name --pr 123 --provider openai --locale Korean Options Option Default Description -p, --pr (required) Pull request number -r, --repo $GITHUB_REPOSITORY Target repository --provider anthropic anthropic | openai | gemini -m, --model provider default Model id (claude-opus-4-8, gpt-5, gemini-flash-latest) --locale English Language for the review output; auto detects it from the PR title/body --paths — Only review files matching these comma-separated globs (monorepo scoping) --max-tokens — Cost guard: stop launching new batches once this many tokens are spent --force — Review even draft, WIP-titled, or skip-review-labeled PRs (skipped by default) --exclude — Comma-separated globs or substrings to skip (added to defaults: lockfiles, dist/, build/, …) --min-severity — Drop findings below this severity (e.g. suggestion hides nitpicks) --fail-on — Exit 1 if any finding is at or above this severity — use as a CI quality gate --context patch full sends complete file contents to the model for better accuracy (more tokens) --event comment auto approves clean PRs and requests changes on critical findings (falls back to comment on your own PRs) --verify off Second model pass that rejects unconfirmed findings --verify-provider same provider Use a separate provider for verification --verify-model provider default Use a separate verification model --verify-failure abort abort, keep, or drop when verification fails --output text json or sarif for machine-readable results --fail-on-incomplete off Fail when filtering, missing patches, or the token budget leaves part of the change unreviewed --check-run off Publish findings as GitHub Check Run annotations --no-dedupe — Repost findings already commented by a previous pr-sage review (dedup is on by default) --no-incremental — Always review the full PR diff instead of only commits since the last pr-sage review --batch-chars 80000 Max diff characters per model request; larger PRs are reviewed in batches --config .pr-sage.json Config file path --dry-run — Print the review to stdout instead of posting Required environment variables: GITHUB_TOKEN (with pull_requests: write), plus the API key for your provider (ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY). On repeat runs (e.g. new commits pushed to the PR), pr-sage reviews only the commits pushed since its last review (incremental mode), skips findings it has already commented, and posts nothing when there is nothing new — no duplicate-comment spam, no wasted tokens. If your repo has a CLAUDE.md or CONTRIBUTING.md, it is automatically injected as review context (disable with "repoContext": false). GitHub Enterprise works out of the box via $GITHUB_API_URL or the githubApiUrl config field. Each run prints its token usage to stderr (LLM usage: N call(s), X input / Y output tokens) so cost stays visible. Every summary also reports review coverage. Partial reviews never auto-approve a PR. Use --fail-on-incomplete when incomplete coverage must fail the CI quality gate. Self-hosted / local models Point the OpenAI provider at any OpenAI-compatible server and private code never leaves your machine — no API key required: ollama pull qwen2.5-coder:14b OPENAI_BASE_URL=http://localhost:11434/v1 \ npx pr-sage review --repo owner/repo --pr 123 --provider openai --model qwen2.5-coder:14b Works the same with vLLM, LM Studio, or any gateway that speaks the OpenAI chat completions API. For GitHub Actions, init --provider self-hosted generates a workflow for a self-hosted runner; localhost must refer to that runner, not a GitHub-hosted VM. Configuration file Put a .pr-sage.json in the directory you run from (CLI flags override it): { "provider": "anthropic", "locale": "auto", "exclude": ["src/generated/", "/*.snap"], "paths": ["packages/web/"], "minSeverity": "suggestion", "failOn": "critical", "context": "full", "maxTokensPerRun": 200000, "failOnIncomplete": true, "skipLabels": ["skip-review"], "verify": true, "verifyProvider": "gemini", "verifyModel": "gemini-flash-latest", "verifyFailure": "abort", "checkRun": true, "pathRules": [ { "paths": ["packages/api/"], "instructions": "Check public API backward compatibility.", "minSeverity": "suggestion", "failOn": "warning" } ], "instructions": "We use Result for error handling — flag thrown exceptions in domain code. Prefer early returns over nested conditionals." } instructions is injected into the review prompt — use it for team conventions the reviewer should enforce. GitHub Action # .github/workflows/pr-sage.yml name: AI Review on: pull_request: types: [opened, synchronize] permissions: contents: read pull-requests: write checks: write concurrency: group: pr-sage-${{ github.event.pull_request.number }} cancel-in-progress: true jobs: review: runs-on: ubuntu-latest steps: # Read .pr-sage.json from trusted base code, never PR-controlled code. - uses: actions/checkout@v4 with: ref: ${{ github.event.pull_request.base.sha }} persist-credentials: false - uses: Kyeom1997/pr-sage@v1 with: provider: anthropic anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} locale: Korean fail-on: critical # optional: block merge on critical findings fail-on-incomplete: "true" check-run: "true" To upload SARIF, add security-events: write and set sarif: "true". The Action also exposes first-class paths, max-tokens, verify-provider, verify-model, verify-failure, and openai-base-url inputs. The base-SHA checkout is deliberate: configuration and repository guidelines must come from trusted base code. The PR diff itself is always fetched through the GitHub API, and pr-sage rechecks the head SHA immediately before posting so a slow review cannot comment on a superseded commit. Measuring it scripts/bench.mjs runs pr-sage (dry, nothing posted) over recent merged PRs of any public repo and records findings, severity mix, latency, and token usage, plus a labeling sheet for computing the valid-review rate: node scripts/bench.mjs --repos fastify/fastify --per-repo 5 --provider gemini --mode recall measures the other axis: it picks merged PRs that received human review comments, reviews each PR's first commit (the state humans reviewed), and reports how many human-flagged locations pr-sage also flags — with a side-by-side sheet for manual verification. The Quality Benchmark workflow can run either benchmark manually and uploads the generated JSON and labeling sheet as workflow artifacts. Measured (2026-07, 28 PRs) 28 recently merged PRs across fastify, hono, GitHub CLI, and Vite (gemini-flash-lite, zero run failures): 25/28 (89%) produced zero comments — quiet on code that had already passed human review. That silence is the point: no noise on clean diffs. The other 3 PRs got 7 findings. Verifying each claim against the actual diff: 3/7 valid overall, 2/3 for warning severity — the noise concentrated in the suggestion tier. Re-running those PRs with --verify kept exactly the 2 diff-confirmed-valid findings (a real GPG-signing regression question in a deployment workflow) and rejected every invalid one. Median 2.3 s and ~3.5 k input tokens per PR (≈ $0.01 for all 28 PRs at flash-lite list pricing). Raw results and the per-finding verification notes live in bench-results/. Caveats: small sample, and merged-PR sampling measures noise, not recall — a detection benchmark (reviewing pre-review commits of PRs that later got human fixes) is future work. How it works Fetches the PR metadata and per-file patches from the GitHub API. Annotates both sides of the diff — added/context lines with new-file numbers, deleted lines with old-file numbers — so findings can anchor to removed code too (e.g. "this deleted validation was still needed"). Lockfiles/build artifacts are filtered out. Asks the LLM for a structured review (JSON schema — no parsing heuristics): summary + findings with path, line, severity, and an optional single-line suggestion. Validates every finding at runtime (zod) and against the diff (GitHub rejects reviews that comment on lines outside the diff), demotes unsafe multi-line suggestions, skips findings already posted by a previous run, retries on provider rate limits, and posts one review: inline comments + summary. Severities: 🔴 critical · 🟡 warning · 🔵 suggestion · ⚪ nitpick. Safe single-line fixes are posted as GitHub suggestion blocks you can commit with one click. Programmatic use import { GitHubClient, createProvider, runReview } from "pr-sage"; const github = new GitHubClient(process.env.GITHUB_TOKEN!, "owner", "repo"); const pr = await github.fetchPullRequest(123); const provider = createProvider("anthropic"); const { result } = await runReview(provider, pr, { locale: "English", exclude: [], batchCharBudget: 80_000, log: console.error, }); Security & privacy Reviewing code means sending diffs (and optionally full files) to the LLM provider you choose — read SECURITY.md for the exact data flow, provider policy links, prompt-injection mitigations, an [truncated for AI cost control]