待翻译：Defusing an 18 GB bomb inside Redis

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Engineering notes Defusing an 18 GB bomb inside Redis 29 August 2026 · #redis · #horizon · #queues · #ops · #incident · #reliability At 07:07:38 UTC on 28 August, for about two seconds, every request into the back office returned a 500. Refund pages, dispatch screens, the chat dock, webhooks — everything, on both application servers, at once. Then it stopped, and the system carried on as if nothing had happened. Two seconds is an interesting duration for an outage. Long enough that 577 requests failed and 29 people saw error toasts; short enough that by the time anyone looked, there was nothing to see. This note is about how we traced it, because the trail is more useful than the incident — it goes through a misleading error bucket, a queue nobody was reading, and eighteen gigabytes of Redis that had no reason to exist. The bucket that pointed the wrong way The report arrived as a frontend Sentry issue: an axios 500, ~1,400 events since June, recent ones on refund pages. The obvious suspect was the endpoint those pages call on mount — a payments table for the refund screen. We read the action, ran its generated SQL against production data, checked the exact invoice in the events. All healthy. The issue turned out to be a catch-all bucket: every axios 500 in the app, from any page, grouped under one stack trace — because the stack trace of "a request failed" is the same regardless of which request failed. The refund-page events were really the staff chat dock's 60-second poll dying underneath whatever page happened to be open. The lesson for anyone triaging frontend Sentry: when the grouped frames are all inside axios/lib/core, the issue tells you that requests fail, never which — go to the traces. The traces led to the backend twin: RedisException: BUSY Redis is busy running a script, thrown from the middleware that queues an audit log of every authenticated request. That middleware runs on everything — which is why everything failed, and why the errors named every page in the app except the one that caused them. Eighteen gigabytes nobody was reading BUSY means a Lua script has been executing for more than five seconds and Redis is refusing all other clients until it finishes. So: which script? The script cache held only the ten standard Laravel and Horizon scripts, 4.7 KB between them, p99.9 of 266 µs over 1.6 billion calls. There was no big script. Something had made a tiny one slow. The answer was one key: a :delayed sorted set holding 6,947,956 jobs — 18.08 GB, in a Redis instance whose entire footprint was 18.1 GB, on a box that was 31 GB into swap and intermittently failing its RDB snapshots. Under that much fork and swap pressure, a routine script only needs to touch a few cold pages to stall past five seconds once. It did, once, and 185 connected clients got BUSY for two seconds. How does a delayed set reach seven million jobs? In June, a non-essential external call — a periodic data refresh against a third-party service — was moved onto its own queue — a reasonable tidy-up — but the new queue name was never added to any Horizon supervisor. No worker ever polled it. An hourly command kept dispatching a few thousand delayed jobs into it, the delayed-to-ready migration only runs when a worker polls, and so the set only ever grew: ~86,000 jobs a day, for eighty days, in a lane nothing was reading. Worth saying plainly: this was a low-traffic refresh lane, a call whose worst failure mode is data going a little stale — nothing user-facing depended on it, and everything around it ran normally the whole time. Degraded, not down; which is exactly why nobody noticed for three months. The things that fail loudest get fixed fastest. The things that fail silently get a blog post. Delete first, deploy second The tempting fix — add the queue to a supervisor, deploy — would have caused a much bigger outage than the one we were fixing. The first worker to poll the queue runs the migrate-expired-jobs script, which moves all due jobs in one atomic Lua call. All 6.9 million were due. That script would have held Redis for minutes, not seconds. So the order mattered, and it went in the incident report in bold: UNLINK the key — not DEL, which frees an 18 GB value synchronously and is the outage; UNLINK hands it to a background thread. Memory fell from 18.1 GB to 1.2 GB in under a minute, and the server climbed out of swap. Deploy the supervisor change, into a now-empty queue. Wrap the audit-log dispatch in rescue(). Request logging is a nice-to-have; the request is not. A Redis hiccup now degrades telemetry instead of failing every page — the same shape as only flushing the cache you changed: blast radius is a design decision. Two aftershocks, both worth their own line. The unique-job locks from all those dispatches had no TTL and only release when a job runs — 2.47 million orphaned lock keys, 99.9% of that database, swept with a SCAN + TTL + UNLINK command that now lives in the repo. And when the queue finally got its worker, every job failed anyway: a morphTo relation whose name didn't match its method name loaded fine lazily but came back null after the queue serializer restored it eagerly. Fixed, plus one more of the same species found by sweeping the models. The lane's first full run in three months went through cleanly the same evening — 4,737 jobs, zero failures. The check that was missing Every piece of this had a guard except the first one. Deploys are gated, tests run against a real database, Horizon's dashboard shows every queue it knows about — and that last clause is the hole. A queue with no supervisor is invisible to the dashboard precisely because nobody claimed it. The system had no way to say "jobs are arriving somewhere nobody is assigned to look". Now it does, and it is embarrassingly small: an hourly command that scans every :delayed and :reserved set in the queue Redis, compares queue names against the supervisor config, and posts to the ops channel when anything unclaimed holds jobs or anything claimed holds too many. It would have fired in the first week of June with a count of a few thousand, and this post would not exist. That is the general rule we keep re-learning, one incident at a time: every buffer needs a reader, and every buffer without one needs an alarm. A queue, a cache, a log table, an inbox — anything that absorbs writes without a consumer is not infrastructure, it is a bomb on a slow fuse — and this one ticked for eighty days before we heard it.