Show HN: Miser – Cost-Optimised AI Gateway
Miser is an open-source, Rust-based AI gateway that routes OpenAI-compatible requests to the cheapest capable model via OpenRouter. It offers heuristic, local-LLM, cloud-LLM, and hybrid classifier modes, preserves streaming responses, and exposes routing metadata. The README covers setup, evaluation, API key management, and hardened deployment.
Notifications You must be signed in to change notification settings
Fork 0
Star 1
BranchesTags
Open more actions menu
Folders and files
NameName
Last commit message
Last commit date
Latest commit
History
42 Commits
42 Commits
.github/workflows
.github/workflows
config
config
crates
crates
deploy
deploy
docs
docs
evals
evals
prototypes/typescript
prototypes/typescript
scripts
scripts
.gitignore
.gitignore
.gitleaks.toml
.gitleaks.toml
Cargo.lock
Cargo.lock
Cargo.toml
Cargo.toml
Dockerfile
Dockerfile
LICENSE
LICENSE
README.md
README.md
deny.toml
deny.toml
Repository files navigation
Miser is an open-source, Rust-based AI gateway that routes OpenAI-compatible requests to the cheapest capable model through OpenRouter.
Documentation
Documentation index
High-Level Design
Low-Level Design
Security Model
Operations Runbook
Evaluation Methodology
Architecture
OpenCode / Codex / Aider / SDK | v Miser Gateway :8787 | override -> structural -> heuristics | local LLM (optional) | cloud LLM (optional) | tier policy -> OpenRouter model
The gateway is stateless, preserves unknown OpenAI request fields, forwards streaming responses, and exposes routing metadata through x-miser-* headers.
Classifier modes
Configure classifier.mode in config/miser.toml:
heuristic: zero-cost, local structural and regex classification
local_llm: OpenAI-compatible Ollama or local endpoint
cloud_llm: OpenAI-compatible cloud classifier
hybrid: heuristics first, then bounded local/cloud fallback
The default hybrid mode is conservative: the low-latency heuristic result is accepted when confident; optional model calls are attempted only for ambiguous requests and have independent deadlines.
Run locally
cp config/miser.env.example .env export OPENROUTER_API_KEY=sk-or-... cargo run -p miser-gateway -- --config config/miser.toml
Configure OpenCode:
{ "$schema": "https://opencode.ai/config.json", "provider": { "miser": { "npm": "@ai-sdk/openai-compatible", "name": "Miser Gateway", "options": { "baseURL": "http://127.0.0.1:8787/v1", "apiKey": "local" }, "models": { "auto": { "name": "Miser Auto" } } } }, "model": "miser/auto" }
Endpoints
POST /v1/chat/completions
GET /v1/models
GET /health/live
GET /health/ready
Evaluation
The versioned corpus is evals/cases.jsonl.
cargo run -p miser-evals -- --mode heuristic
The evaluator reports exact and adjacent-tier accuracy plus a confusion matrix. Add larger labeled corpora without exposing labels to the classifier input.
VPS benchmark
The Rust gateway was evaluated on the deployed VPS on 2026-08-09:
Strategy Hardware Cases Exact Adjacent Under-route Failures p50 latency p95 latency
Rust heuristics 2 vCPU, 7.8 GiB RAM, no GPU 25 92.0% 92.0% 0.0% 0
Create a user API key:
curl -X POST https://miser.rajeev.me/admin/keys \ -H "Authorization: Bearer miser_admin_" \ -H "Content-Type: application/json" \ -d '{"owner": "your-name"}'
List keys:
curl https://miser.rajeev.me/admin/keys \ -H "Authorization: Bearer miser_admin_"
Delete a key:
curl -X DELETE https://miser.rajeev.me/admin/keys/{key_id} \ -H "Authorization: Bearer miser_admin_"
Using API keys
{ "provider": { "miser": { "npm": "@ai-sdk/openai-compatible", "name": "Miser Gateway", "options": { "baseURL": "https://miser.rajeev.me/v1", "apiKey": "miser_" }, "models": { "auto": { "name": "Miser Auto" } } } }, "model": "miser/auto" }
Keys are validated on every request using constant-time hash comparison. The raw key is returned only once at creation time.
Deployment
The included Dockerfile creates a non-root image. deploy/miser.service provides a hardened systemd unit. Copy config/miser.toml and a mode-600 environment file containing OPENROUTER_API_KEY to the server.
Prototype
The original Bun/TypeScript prototype is preserved under prototypes/typescript for comparison and migration reference.
Development
cargo fmt --all cargo check --workspace cargo test --workspace cargo clippy --workspace --all-targets -- -D warnings
License
MIT
miser
MIT license
Security policy
Security policy
Activity
Stars
1 star
Watchers
0 watching
Forks
0 forks
Report repository