Why Your Production RAG System Slowly Gets Worse

Jun 16, 2026

Why Your Production RAG System Slowly Gets Worse

Background

Production RAG systems rarely fail through a single catastrophic event. More commonly, reliability erodes through a sequence of operational changes: documentation evolves, retrieval behavior shifts, prompts are revised, dependencies change, and evaluation datasets become stale.

Traditional engineering practices classify failures by system components—retrievers, prompts, vector databases, or language models. While useful for implementation, this perspective provides limited guidance for operating production AI systems over time.

This article proposes a reliability framework based on three complementary dimensions:

Failure Dynamics — how reliability changes over time

Reliability Control Surface — where engineers can observe and intervene

Detectability — how easily the failure is discovered before users are affected

To illustrate the framework, a controlled experiment simulates seven weeks of gradual documentation evolution in a production-style RAG system. The experiment demonstrates one representative failure class—Gradual Knowledge Drift—and shows why this class of failure frequently escapes traditional operational monitoring.

1. Introduction — AI Systems Rarely Fail the Way Traditional Software Does

Modern software systems fail in ways that operations teams understand well. A bad deployment increases error rates. A database outage causes requests to fail. A networking issue adds latency. Infrastructure becomes unavailable. These failures are disruptive, but they are also highly visible. Dashboards turn red, alerts fire, and engineers know where to start investigating.

Retrieval-Augmented Generation (RAG) systems introduce a different class of failure. Usually , a production RAG application can appear perfectly healthy from an operational perspective. Requests complete successfully, APIs return HTTP 200 responses, latency remains within service-level objectives, and every component in the architecture is online. Traditional monitoring tools report a healthy system. Yet users begin to lose confidence in the answers.

Fundamentally, we are trying to solve the AI reliability problem instead of the traditional software reliability problem.

Figure 1 - Traditional Software Reliability vs AI Reliability Timeline

From the graph, the key differences is that traditional software failures are around discrete events and gives immediate feedback; while RAG systems degrades gradually and usually invisible to infrastructure-level monitoring. Fundamentally, traditional software’s reliability is typically judged by correctness and availability: either the service works or it doesn’t. RAG systems add another dimension—knowledge quality. A system can achieve excellent uptime while steadily becoming less reliable.

This reframes reliability from a problem of system correctness to a problem of sustained knowledge quality.

1. Why Existing Classifications Are Insufficient

What do we know about RAG system failures. Perhaps newly published documentation isn’t being retrieved. Maybe document metadata has drifted, reducing retrieval accuracy. An embedding model has changed, but only part of the corpus has been re-indexed…

Current discussions usually classify failures by components, some of the examples are :

ComponentTypical failures

Embedding modelPoor semantic representations, embedding drift after model changes, domain mismatch, multilingual mismatch

Vector databaseLow recall, indexing errors, stale or missing vectors, incorrect filtering, ANN search inaccuracies

ChunkingChunks too large/small, broken context boundaries, duplicated information, loss of semantic coherence

RetrieverIrrelevant documents retrieved, low recall, poor ranking, metadata filtering mistakes

RerankerRelevant documents demoted, irrelevant documents promoted, unstable ranking

PromptHallucinations, ignored context, prompt injection, poor instruction following, format inconsistencies

LLM / GeneratorHallucination, incorrect synthesis, unsupported claims, reasoning errors, overconfidence

Knowledge baseOutdated documents, incomplete corpus, inconsistent information, stale data

Ingestion pipelineFailed indexing, partial ingestion, parsing/OCR errors, metadata extraction failures

Figure 2 - AI Failure Examples

These do explain where failures originate. However, they hardly explain:

how failures evolve

when engineers discover them

which operational strategy is appropriate

Production RAG system operations require a reliability model, not only an architecture model.

1. A Reliability Framework for Production AI Systems

Imagine an engineer receiving the following incident report:

“The RAG system is hallucinating more than usual.”

Although the statement describes a symptom, it immediately raises several unanswered questions.

Has the system failed suddenly after a deployment, or has answer quality been declining for weeks? Is the root cause likely to be in the knowledge base, the retrieval pipeline, or the generation stage? Should engineers inspect operational dashboards, rerun evaluation suites, or begin a deeper investigation?

The difficulty is not a lack of observability—it is a lack of structure for reasoning about production AI failures.

From examining recurring production incidents, I found that most failures can be described along three complementary dimensions:

Failure Dynamics describe how reliability changes over time.

Reliability Control Surfaces identify where corrective action is most effective.

Detectability characterizes how easily the failure is discovered before affecting users.

Rather than treating every incident as unique, these dimensions provide a common language for understanding, classifying, and responding to production AI failures.

Dimension 1 — Failure Dynamics

When a RAG incident occurs, the first question engineers should ask is not what failed, but how reliability changed over time.

Traditional software systems are typically designed around discrete failures. A deployment introduces a regression, a dependency fails, or a resource becomes exhausted. Reliability changes are usually tied to identifiable events, allowing engineers to reason about incidents as immediate failures.

Production RAG systems behave differently. Reliability often changes continuously rather than discretely. Documentation evolves, retrieval behavior shifts, prompts are revised, and evaluation datasets become stale. Individually, these changes appear harmless; collectively, they reshape the behavior of the system. As a result, understanding a production AI incident begins with a different question:

How did reliability evolve over time?

This leads to the first dimension of the framework: Failure Dynamics.

Immediate Immediate failures appear immediately after a discrete system change or unexpected input. They are typically associated with deployments, prompt revisions, tool misconfiguration, or invalid context injection. Engineers usually observe an immediate drop in correctness or task completion

Gradual Gradual failures emerge through a sequence of individually harmless changes. Documentation evolves, retrieval behavior shifts, evaluation datasets become stale, or models are upgraded incrementally. No single change is sufficient to trigger an incident, but their cumulative effect steadily erodes reliability.

Threshold Threshold failures remain latent until accumulated changes push the system beyond a critical operating boundary. Reliability appears stable until a tipping point is reached, after which performance degrades abruptly.

Oscillating Oscillating failures exhibit inconsistent reliability under similar operating conditions. Performance alternates between successful and unsuccessful outcomes because the underlying system behavior depends on input distribution, retrieval ordering, model stochasticity, or changing operational conditions.

Cascading Cascading failures originate from a local defect that propagates through downstream workflow stages. A retrieval error may influence planning, which affects tool selection, memory updates, and ultimately produces a significantly larger end-user failure than the original defect alone.

Dimension 2 — Reliability Control Surface

Once the failure dynamics have been identified, the next engineering question is:

Where should engineers intervene?

Failure Dynamics describe how reliability changes. Reliability Control Surfaces describe where reliability can be observed, influenced, and improved.

In traditional software systems, the answer is often localized. Engineers scale infrastructure to address resource contention, upgrade dependencies to resolve compatibility issues, or adjust service-level trade-offs between latency, availability, and consistency. The intervention point is usually well-defined because the system itself is deterministic.

Production RAG systems are different. A single user-visible failure may emerge from interactions across multiple stages of the pipeline. Corrective actions therefore require engineers to identify the control surface where reliability can be most effectively improved.

We define five primary Reliability Control Surfaces.

Knowledge The knowledge surface governs the quality of the information available to the system. Engineers intervene here by improving the corpus itself: removing stale documents, eliminating duplicates, correcting inconsistencies, or refining document organization. If the system retrieves incorrect knowledge, no downstream component can reliably recover the correct answer.

Retrieval The retrieval surface determines which knowledge reaches the model. Engineers adjust retrieval algorithms, chunking strategies, embedding models, metadata filters, rerankers, and search parameters to improve the relevance and completeness of retrieved context.

Generation The generation surface governs how retrieved context is transformed into an answer. Prompt design, model selection, decoding strategies, and structured output constraints all influence whether the model produces accurate, complete, and faithful responses.

Evaluation The evaluation surface determines how reliability is measured and enforced. Rather than improving answers directly, evaluation establishes quality gates through automated benchmarks, regression tests, and production monitoring. It answers the question: Has reliability changed enough to require intervention?

Operations The operations surface coordinates how the entire system behaves in production. Version management, deployment policies, rollout strategies, monitoring, traffic routing, and incident response all influence the long-term reliability of the application, even when individual components remain unchanged.

Dimension 3 — Detectability

The previous dimension answered where engineers should intervene. Detectability answers a different operational question:

How likely is this failure to be discovered before users experience it?

Not all failures are equally visible. Some immediately trigger monitoring systems, while others remain hidden behind apparently successful requests and fluent model responses. From an operational perspective, the cost of a failure depends not only on its severity but also on how long it remains undetected.

Traditional software systems have benefited from decades of investment in observability. Infrastructure failures, resource exhaustion, deployment regressions, and service interruptions typically produce measurable signals that monitoring systems can detect automatically.

Production AI systems introduce a different class of reliability problems. A request may complete successfully, latency may remain stable, and no infrastructure alarms may fire, yet answer quality can still deteriorate. In these cases, correctness—not availability—becomes the primary operational concern.

We therefore classify production AI failures according to their

[truncated for AI cost control]