Show HN: TraceGen – realistic OpenTelemetry traces, incl. AI-agent, one binary

Uh oh!

There was an error while loading. Please reload this page.

Notifications You must be signed in to change notification settings

Fork 0

Star 2

BranchesTags

Open more actions menu

Folders and files

NameName

Last commit message

Last commit date

Latest commit

History

51 Commits

51 Commits

.claude

.claude

.github

.github

.img

.img

cmd/tracegen

cmd/tracegen

.gitignore

.gitignore

.goreleaser.yml

.goreleaser.yml

CONTRIBUTING.md

CONTRIBUTING.md

DOCKERHUB.md

DOCKERHUB.md

Dockerfile

Dockerfile

LICENSE

LICENSE

README.md

README.md

WHERE-TRACEGEN-RUNS.md

WHERE-TRACEGEN-RUNS.md

go.mod

go.mod

go.sum

go.sum

Repository files navigation

A single-binary distributed trace generator that produces realistic, topology-rich OTLP traces and correlated logs. No Docker Compose, no microservices to deploy, no infrastructure - just one executable (or a 5.7 MB container) that simulates a full e-commerce platform with up to 28 services, dozens of pods that scale with the topology, and 40 scenario flows - including AI agentic scenarios with full OTel GenAI semantic conventions. Three complexity levels (light/normal/heavy) let you scale from a clean 10-service demo to the full topology with AI.

Built for testing observability platforms, load testing trace pipelines, and showcasing distributed system visualizations - for both traditional APM and LLM observability.

Why This Exists

Every existing trace generator falls into one of two categories:

Flat span generators (telemetrygen, tracepusher) - produce uniform, identical spans with no service topology

Full demo apps (OTel Astronomy Shop, Jaeger HotROD) - require Docker Compose with 15+ containers and ~6GB RAM

And none of them generate AI agentic traces. The LLM observability market has no standalone tool that combines traditional APM with LLM observability. Every specialized LLM tool (Langfuse, LangSmith, Helicone, Arize, Traceloop, Portkey, Galileo) tracks token usage, model costs, and agent tool calls - but none of them provide traditional distributed tracing.

This tool generates topology-rich, failure-injectable traces from a single binary - covering both traditional microservice flows AND AI agentic patterns with OTel GenAI semantic conventions. One binary proves that a platform can visualize both.

Quick Start

Download the latest release (or build from source)

go install github.com/ImmersiveFusion/if-opentelemetry-tracegen/cmd/tracegen@latest

Send to a local OTLP collector (Jaeger, Tempo, etc.)

tracegen -insecure

Send to a remote endpoint with auth headers

tracegen -endpoint your-otlp-endpoint:443 -headers "api-key=YOUR_KEY"

Or set headers via the standard OTel environment variable

export OTEL_EXPORTER_OTLP_HEADERS="api-key=YOUR_KEY" tracegen -endpoint your-otlp-endpoint:443

See it in 3D - Send traces to IAPM (tracegen -endpoint otlp.iapm.app:443 -headers "api-key=YOUR_KEY") to explore them as a 3D force-directed graph, drill into conventional trace waterfalls for detailed analysis, and get AI-assisted insights from Tessa. For a ready-made example without any setup, try the OpenTelemetry Chaos Simulator at demo.iapm.app - a fully interactive sandbox with visual failure injection.

Live demo grids — see it running

Seven always-on demo grids stream live OpenTelemetry traces into IAPM's 3D player right now — a clean baseline, an AI-native app, a blended environment, phantom-service detection, an AI-outage, and a full incident. Each grid is this container, deployed declaratively via GitOps (Argo CD) in the Immersive Fusion cloud — multi-arch and distroless, one matrix row per grid, shipping to otlp.iapm.app:443.

See them in 3D: the full experience is the IAPM 3D client — install it and open a grid to walk the live traces. On mobile or can't install right now? IAPM Web runs the same grids in your browser at portal.iapm.app.

Where else does TraceGen run? → — a community board of deployments. Add yours.

Features

28 Microservices

Traditional Services (20)

Service Pods Role

web-frontend 2 Browser client, SPA

api-gateway 3 HTTP routing, auth

order-service 3 Order lifecycle

payment-service 2 Stripe integration

inventory-service 2 Stock management

notification-service 2 Event-driven notifications

user-service 2 Auth, profiles

cache-service 3 Redis cluster

search-service 2 Elasticsearch queries

scheduler-service 1 Cron jobs (singleton)

auth-service 3 JWT, webhook verification

recommendation-service 2 ML-based recommendations

cart-service 2 Shopping cart

product-service 3 Product catalog

shipping-service 2 Rates, labels, tracking

fraud-service 2 ML fraud scoring

email-service 2 SMTP relay (SendGrid)

tax-service 1 Tax calculation

analytics-service 3 Event tracking (Kafka)

config-service 1 Feature flags

AI Services (8)

Service Pods Role

llm-gateway 3 OpenAI API routing, token tracking

embedding-service 2 Text-to-vector operations

vector-db-service 2 Qdrant similarity search

ai-agent-service 2 Agent orchestration (plan/act/reflect)

content-moderation-service 2 Safety classifiers, PII detection

model-registry-service 1 Model versioning (singleton)

feature-store-service 2 ML feature serving

data-pipeline-service 2 Batch embedding, retraining

All 59 pods are distributed across 5 AKS VMSS nodes (2 node pools) with realistic service.instance.id and host.name resource attributes.

40 Scenario Flows

Traditional Scenarios (15 original + 13 new)

Scenario Graph Shape Key Pattern

Create Order Long chain (8 services, 14+ spans) Producer/consumer with queue delays

Search & Browse Linear with cache Elasticsearch + Redis

User Login Branching (success/failure) Auth with session creation

Failed Payment Error chain Stripe 402 + error propagation

Bulk Notifications Fan-out (3-5 parallel) Batch email processing

Health Check Star topology (6 parallel) Concurrent health pings

Inventory Sync Fan-out + reindex Parallel cache warming

Scheduled Report Headless chain (no UI) Cron job entry point

Stripe Webhook Headless chain (no gateway) External callback entry

Recommendations Scatter-gather / bowtie Fan-out to 3, gather, cache

Add to Cart Cross-service with feature flags Config service + analytics

Full Checkout Monster chain (15 services) Tax+shipping parallel, fraud ML

Shipping Update Carrier webhook (headless) External webhook + email relay

Saga Compensation Forward chain + 4-way compensation fan-out Payment retries + rollback

Timeout Cascade Branching with circuit breaker Stale cache fallback

User Registration Linear with async branch Email verification token, duplicate detection

Product Review Write + async moderation Optimistic write + background processing

Return/Refund Parallel reverse flow (16-18 spans) Parallel refund + restock, reverse money flow

Wishlist + Price Alert Write-through with async Write-through cache, async price monitoring

Coupon Application Validation chain Cart recalculation, validation branch

Gift Card Purchase Payment splitting Balance check, payment splitting

Subscription Management Webhook-driven lifecycle Stripe subscription, renewal webhook

A/B Test Exposure Feature flag branch Variant assignment, sticky session

Rate Limiting Early termination (4-6 spans) Redis sliding window, 429 response

Admin Product CRUD Write-amplification fan-out Cache + search reindex on write

Order History Paginated read Keyset pagination, cursor-based

Support Ticket Cross-domain trace SLA assignment, team routing

Multi-Currency Checkout External API chain FX rate API, cache hit ratio

AI Agentic Scenarios (12)

Scenario Graph Shape Key Pattern

Semantic Search (RAG) Linear with 2 LLM calls (14-16 spans) Embedding + vector search + LLM reranking

AI Chatbot with Tool Use Double bowtie (18-22 spans) Plan -> fan-out tool calls -> synthesize

AI Content Moderation Parallel classifiers + 3-way branch (12-16 spans) Safety/spam scoring, guardrail decisions

Multi-Step Agent Iterative loop (28-40 spans) Plan -> act -> reflect cycle (3-5 iterations)

AI Customer Support Branching with escalation (16-20 spans) Sentiment classification, intent detection

AI Content Generation Linear with safety filter (12-15 spans) Temperature-controlled generation, content safety

Embedding Pipeline High fan-out batch (25-40 spans) Batch chunking, parallel embedding, vector upsert

Dynamic Pricing Agent Headless agent (14-18 spans) Feature store lookup, autonomous price updates

Fraud with Explainability Linear with LLM explanation (10-12 spans) SHAP-style feature attribution via LLM

Inventory Reorder Agent Autonomous agent (16-20 spans) Demand forecast, autonomous purchase orders

Model Retraining Pipeline Batch pipeline (14-18 spans) ML training spans, model registry, quality gate

Conversational Commerce Multi-turn session (10-14 spans/turn) Growing context tokens, session continuity

Note: Failed Payment, Saga Compensation, Timeout Cascade, lost messages, and retry storms only activate when -errors > 0. AI error scenarios (rate limits, hallucinated tool calls, token budget exceeded, content filter blocks) also require -errors > 0.

Correlated Logs

Every service emits OTel log records via OTLP alongside traces. Logs are automatically correlated with the active span context (trace_id, span_id), so your APM platform can link logs to the exact span that produced them.

ERROR logs are emitted alongside every exception event (cache failures, DB errors, payment declines, LLM rate limits, agent failures)

WARN logs fire on auth failures, content moderation flags, payment retries, and LLM fallbacks

INFO logs cover request entry points, payment processing, fraud analysis, agent invocations, and iteration progress

Disable with -no-logs to emit traces only.

OTel GenAI Semantic Conventions

All AI scenarios emit spans following OTel GenAI Semantic Conventions and matching the exact span shapes produced by Microsoft Semantic Kernel and Microsoft Agent Framework.

Span types:

Span Name Pattern SpanKind Example

chat {model} CLIENT chat gpt-4o

embedding {model} CLIENT embedding text-embedding-3-small

invoke_agent {name} CLIENT invoke_agent CustomerSupportAgent

execute_tool {name} INTERNAL execute_tool get_order_status

{operation} {collection} CLIENT query product-embeddings

Attributes on every LLM span:

gen_ai.system - LLM provider (e.g., openai)

gen_ai.request.model / gen_ai.response.model - model requested and used

gen_ai.usage.input_tokens / gen_ai.usage.output_tokens - token consumption

gen_ai.response.finish_reasons - completion reason (stop, tool_calls, length, content_filter)

gen_ai.response.id - unique response identifier

gen_ai.request.temperature, gen_ai.request.max_tokens - request parameters

Agent-specific attributes:

gen_ai.agent.id / gen_ai.agent.name / gen_ai.agent.description - agent identity

gen_ai.conversation.id - session linking for multi-turn interactions

gen_ai.tool.name / gen_ai.tool.type / gen_ai.tool.call.id - tool call tracking

gen_ai.data_source.id - RAG data source identifier

gen_ai.request.embedding.dimensions - embedding dimensions

These attributes match what every LLM observability tool on the market tracks - enabling direct comparison of visualization capabilities.

Chaos & Failure Injection

Feature Description

Lost messages 5% chance per queue hop that the consumer never fires - trace ends abruptly

Dead consumer mode -no-consumers flag: producers fire, consumers never pick up

Retry storms Payment retries 3x with exponential backoff before saga compensation

Timeout cascades Search service times out, gateway returns 504, circuit breaker serves stale cache

Saga compensation Payment fails after order+inventory committed - triggers 4-way parallel rollback

LLM rate limits OpenAI 429 with token budget details, fallback to text search

Hallucinated tool calls Agent requests non-existent tool, triggers error handling

[truncated for AI cost control]