Stoke – Kill switch for runaway AI agents (Rust, budget caps)

Stoke — your agents, your machines, your rules

Enforce, don't just observe.

Your agents. Your machines. Your rules.

Route to your own hardware — or the cloud, when you choose. Refuse anything over budget. Kill loops in seconds. One ~5.5 MB Rust binary that decides before the money is spent, not a dashboard that reports it after.

Install in 60 seconds ★ Star on GitHub

Works with any OpenAI-compatible agent — point base_url at localhost:8787/v1. Claude Code connects via ANTHROPIC_BASE_URL. Pre-release · MIT · dogfooded daily.

stoke :8787 — illustrative

route it fail over cap it kill it

The problem

Your AI bill is a postmortem.

An agent that starts looping at 2 a.m. keeps spending until someone wakes up. The billing alert is a receipt, not a control — it fires after the damage. Dashboards chart what happened. Stoke decides what is allowed to happen.

Loops don't get tired

A retrying agent will happily replay the same failing request all night, on your metered key.

Alerts arrive late

Spend notifications trail actual usage. By the time one fires, the budget is already gone.

Dashboards observe. Stoke refuses.

An over-budget or looping request never reaches the provider. Refused at the gateway, not flagged in a chart.

Model

Tokens per request 4,000

Requests per minute in the loop 30

Unattended overnight burn (8h)

$0

A runaway loop doesn't sleep. Stoke caps it at your number, or trips the loop breaker in seconds.

Cap it →

Enforcement

Refused at the request path.

Limits that reject requests before a provider ever sees them — not alerts for a human to find in the morning.

1

Hard budget caps, in dollars

Per-key spend caps in USD, set in a [[keys]] config table (each key must also be listed in STOKE_API_KEYS) and applied to the budget guard at startup. Checked before any provider call: over the cap, the request is refused with 429 Budget exceeded — an error your agent can handle, not a log line for later. Live spend per key at /v1/budget; per-request cost rides along in a stoke_cost field on non-streaming responses. Streamed responses accrue spend too — Stoke reads the usage the provider reports as the bytes pass through, and charges when the stream ends or the client hangs up. A metered provider that reports nothing is billed from an estimate, flagged as one, never as $0.

2

Loop kill switch

A two-layer circuit breaker: exact prompt-hash matching plus opt-in semantic similarity, so a loop that rephrases itself still counts as a loop. Five similar requests within 60 seconds blocks the key for 120 seconds. Thresholds are global constants today — per-key tuning is on the roadmap.

3

Per-key rate limits

A sliding 60-second window per API key, set as rate_limit_rpm in the same [[keys]] table and applied at startup. One misbehaving agent gets refused (429); everyone else keeps working.

4

Fail-closed, by architecture

No API keys configured and no dev flag: every request is rejected. Every endpoint except /health requires auth — status endpoints included, because model inventory and node load are reconnaissance data. Federation polling authenticates too. Misconfigured means closed, not open.

Local-first

Run your own models. On every machine you own.

Stoke is dogfooded daily on a real MacBook + Mac Studio + Ollama Cloud setup. Point it at your Ollama nodes and it routes like it knows them — because it polls them.

Node-aware routing

A background registry polls each node's /api/tags (models pulled) and /api/ps (models loaded in RAM). Placement prefers warm nodes, balances the same model across machines by live in-flight count, breaks ties with a latency EWMA, and fails over automatically. Unreachable nodes are excluded until they come back.

Every live stream teaches it: time-to-first-token and tokens/sec are measured per model per node, and context windows plus tool-calling support come from /api/show — so predictions come from your hardware, not guesses.

Federation: Stoke behind Stoke

Provider type "stoke" lets one gateway front another. The peer's models, warm state, live load, and measured speeds are imported through its /v1/nodes — richer state than raw Ollama can report. Forwarded requests carry x-stoke-hop; depth is exactly one, and a deliberate A-to-B-to-A cycle cannot loop — a shipped test script proves it.

The practical win: Ollama stays bound to 127.0.0.1. Stoke is the only network-facing door, and it authenticates.

Auto-routing that optimizes, not guesses

Point your agent at the auto model and Stoke scores every eligible (model, node) pair on estimated cost, predicted latency, and your stated preference order — then picks. auto-cheap weights spend, auto-fast weights speed. Models come only from your config and your discovered nodes; Stoke ships with zero model names.

Hard exclusions, not vibes: prompts that don't fit a model's context window and tool-calling requests aimed at models that can't emit structured tool_calls are ruled out with the reason on the receipt.

Hedged dispatch

Opt-in (hedge = true): small prompts whose model sits warm on two of your machines are fired at both — first past prefill wins, the loser is dropped. Your idle hardware races for you; duplicate local compute buys tail latency that no single node can.

Zero-marginal nodes only — hedging never doubles a cloud bill.

Every decision shows its reasoning — and the road not taken

Auto-routed responses carry the full receipt: the chosen candidate with estimated cost and predicted latency, every alternative and why it lost, and an honest counterfactual — the list price your configured cloud model would have charged. GET /v1/budget aggregates them into a running ledger. Estimates, clearly labeled; never a quality claim.

"stoke_route": { "node": "mac-studio", "auto": { "mode": "auto", "class": "Code", "chosen": { "model": "your-35b", "node": "mac-studio", "warm": true, "est_cost_usd": 0.0, "predicted_ms": 2100 }, "not_taken": [{ "model": "your-8b", "node": "macbook", "warm": false, "est_cost_usd": 0.0, "predicted_ms": 9800 }], "counterfactual_usd_est": 0.0214 // what your configured cloud model would have charged }}

The control point

Every call passes through your pipeline.

This is the actual order of operations for one request. Every stage before the provider call can refuse it — and four of them are yours to write.

AuthenticationFail-closed. No key configured and no dev flag means nothing is served.

can refuse

Budget · rate limit · loop breakerHard caps and the semantic circuit breaker, checked before a provider is ever contacted.

can refuse

Route profilePer-endpoint model, routing mode, and plugin chain.

config

Pre-request hooksRewrite the model or routing — or block the request outright.

yours

Prompt harnessInject a system prompt per task class before the model sees it.

yours

Filter & redactBuilt-in PII redaction strips keys, tokens, emails. Your filters can block on policy.

yours

CacheExact hash, plus opt-in semantic match. A hit never reaches a model at all.

short-circuit

PlacementWarm-first across your machines, scored and explained, with automatic failover.

routes

↓ the provider call — the only step that leaves your gateway

Post-response hooksFormat, transform, and append to a JSONL audit log.

yours

Cost & receiptsPer-key spend recorded, and the decision written back into the response.

observes

The yours stages take webhooks in any language — a plain HTTP endpoint is a plugin. JS/TS plugins run on an embedded V8 behind a compile-time flag. Nothing is a black box: every stage above the provider call can say no, and the ones that transform your prompt run before it leaves the machine.

Who it's for

Three ways people run Stoke.

Coding agents on metered keys

Cap the burn.

You run coding agents — Claude Code, OpenCode, aider, Codex — against metered API keys, and one bad loop is a four-figure morning. Stoke puts a loop kill switch, a rate limit, and a hard USD cap between the agent and the key. The loop kill switch and rate limit apply to every request, streaming included — so a runaway loop is stopped no matter how it talks. OpenCode works end-to-end today; any OpenAI-compatible agent connects via base_url.

Your own GPUs, several machines

One endpoint for all your hardware.

A MacBook here, a Mac Studio there, maybe a Linux box with a GPU. Stoke fronts them all: warm-aware placement, live load balancing of the same model across machines, automatic failover, and federation between gateways — while Ollama never gets exposed to the LAN.

On the roadmap — talk to us

A control plane for your AI product.

You ship an AI product and need usage entitlement enforcement: hard per-tenant caps your billing team can trust, enforced at the request path — a control plane over every customer's LLM spend. This is where Stoke is headed, not shipped yet; we are looking for design partners.

Open a GitHub discussion →

Honest scope: subscription plans are quota-world.

If your agents run on a subscription plan — Claude Max or Pro, ChatGPT Plus — there is no per-request dollar price, so there is nothing to dollar-cap. Stoke can still rate-limit that traffic and kill loops in it, but it cannot cap it in dollars, and we won't pretend otherwise. Hard USD caps apply to metered API keys, where every request has a price.

Sovereignty

Your prompts never leave your infrastructure.

Data residency by architecture, not by contract. An EU-based project, built for people who read data-processing agreements and would rather not need them.

Local by default

Requests route to machines you own. Nothing goes to a cloud provider unless you explicitly configure one. Once configured, automatic failover may use it — so if prompts must never leave your machines, simply don't configure a cloud tier.

One authenticated door

Ollama stays bound to 127.0.0.1. Stoke is the only network-facing surface, and every endpoint except /health requires auth.

Fail-closed status surface

Model inventory and node load are reconnaissance data, so even the status endpoints authenticate. No keys configured means no service — not open service.

Compared honestly

Where Stoke wins. And where it doesn't.

LiteLLM, Portkey and Helicone are good tools. They cover far more providers and go much deeper on observability. Stoke is built for a different job: enforcement.

Stoke LiteLLM Portkey Helicone

Primary job Enforcement — refuse bad requests at the gateway Universal gateway Gateway + observability Observability

Loop kill switch Exact-hash + semantic circuit breaker — — —

Fail-closed default No keys configured = all requests rejected Configuration-dependent Configuration-dependent Configuration-dependent

Node-aware local routing Warm state + live load via /api/ps — — —

Gateway federation Stoke-behind-Stoke, hop-guarded, cycle-safe — — —

Deployment One ~5.5 MB Rust binary, zero runtime deps Python service (database needed for key management features) Hosted, or self-host stack Hosted, or self-host stack

Provider coverage OpenAI-compatible + Ollama + federation. They win here. 100+ providers Broad Broad

Observability depth Per-response route + cost, audit log. Deliberately minimal — they win here. Deep Deep dashboards Their core strength

No benchmark numbers in this table, on purpose. When we publish performance claims, they will ship with a reproducible harness first.

Install

One line, then point your agent at it.

Static binaries for macOS and Linux are built from every commit on main and published to a rolling nightly build — no Rust toolchain needed. Honest status: that rolling build is what exists today; the first tagged stable release is still to come, and the installer falls back to compiling from source on any platform we don't ship.

prebuilt binary, checksum-verified

curl -sSf https://stokegate.com/install | sh -s -- --version nightly

cp stoke.example.toml stoke.toml # your nodes, your providers STOKE_API_KEYS=team-key stoke

Prefer to bui

[truncated for AI cost control]