待翻譯：We measured whether our agents follow their instructions

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Why we started measuring Four separate times, a Firekeep capability existed, worked, and was never used — until one paragraph landed in the rendered instruction block with a concrete, observable trigger. A clarification board that tool descriptions alone never fired. Recall triggers that sat idle while storage and retrieval worked perfectly. A feedback signal the whole ranking layer depended on, living only in a docstring. A calibration system with no predictions in it, because nothing told agents to state their confidence. The pattern was too consistent to ignore: the binding constraint on an agent system is not the tools — it is the instruction layer. And that layer is written by hand, on judgment, and never measured. If a paragraph of instructions is what makes or breaks a capability, then the paragraph deserves what every other production system gets: monitoring. The first table Firekeep records every session as a structured replay trace — memory reads, writes, session lifecycle, gateway decisions — and computes quality metrics per session from that record. Compliance is then a deterministic predicate over those stored metrics. No model in the loop, no judgment calls: memory_read_count > 0 either held for a session or it did not. Here is the founding measurement, computed 2026-08-11 on our own live deployment, over the 32 sessions in the evaluation window: Instruction (as rendered to agents)Predicate over the session recordCompliance "Recall before you answer"memory_read_count > 018/32 — 56% "Write as you go" (record learnings)memory_write_count > 015/32 — 47% Recalled knowledge visibly usedrecall_used_rate > 08/32 — 25% Working state kept currentcontext_snapshot_count > 020/32 — 62% "Declare consequential actions"brier_score is not None0/32 — 0% Outcome-bearing events ≥ 2outcome_event_count >= 210/32 — 31% Read it plainly: barely half of sessions recalled team memory before answering. A quarter visibly used what they recalled. And the instruction to declare consequential actions — shipped the day before with a fresh client release — had reached zero sessions, which is not disobedience but a rollout fact, and the difference matters. This table now lives on the product dashboard, recomputed live, with the predicates frozen so every future number stays comparable to this baseline. What a number like this can and cannot claim A compliance rate measures behavior — whether sessions did the instructed thing. It does not measure whether doing it helped. Our outcome signal is still too thin to support quality claims (near-zero recorded failures, roughly one outcome-bearing event per typical session), so any statement like "recalling first improved results" would currently be fiction, and the dashboard says so on the same surface that shows the rates. The way out of that limitation is recursive, and it is why the 0% row matters most: the instructions most worth enforcing first are the ones that create outcome signal. Feedback compliance populates per-memory usefulness. Declared predictions populate calibration scores. Completion discipline populates real success and failure. Every compliance win makes the quality question more answerable. The measurement caught its own authors first Within a day of shipping the table, an external review filed six findings against it. We fixed four the same evening. Then, while building the next round, we discovered that one of the review's "confirmed" findings was itself wrong — the reviewer had reasoned from call sites without checking a category gate between the emitter and the scorer, and our fix had dutifully relabeled a row to say the opposite of what the code does. The row was measuring genuine agent discipline all along; our correction had overclaimed in the other direction. Both the finding and the counter-finding are recorded, dated, in the design spec. The second catch was worse. The "declare consequential actions" experiment was believed to reach agents through two delivery channels: the rendered instruction file and the MCP handshake. Tracing the receiving end showed the second channel never existed — the local gateway discards backend handshake text and serves its own, which did not carry the instruction. Our test suite asserted the server sends the text; nothing ever asserted an agent receives it. The instruction layer had a silent delivery failure, and no measurement could have caught it, because exposure was not being measured. If your agent instructions have never been measured, this is the shape of what you don't know: not just "are they obeyed" but "did they even arrive". Ours didn't, on one channel, for two days — and we only know that precisely because we started measuring. A rate is only as honest as its denominator That delivery failure set the design for round two. A compliance rate over "all sessions" conflates three different populations: sessions that received the instruction and complied, sessions that received it and didn't, and sessions the instruction never reached — older clients, hand-edited instruction files, runtimes with different delivery surfaces. Counting that third group against the instruction blames the text for the postal service. So sessions now carry exposure receipts. The rendered instruction block is stamped with a content hash; the client re-hashes what is actually on disk at session start — not what the installer intended, what is there — and reports it, along with the runtime and client version, as attribution labels on the session record. The compliance table then splits every rate three ways: exposed (a verified artifact carrying that instruction reached the session), not exposed (affirmatively absent), and unknown — and everything unverifiable is unknown, never counted against anyone. Every session that predates the receipts reads unknown, forever. Nothing backfills, and the dashboard discloses it. The pre-registered experiment, and its first movement The 0% row doubles as a natural experiment, pre-registered in the design spec: the instruction shipped with a recorded 0/32 baseline, so the brier_score row moving off zero — or not moving — is the first Living Instructions measurement, with the spec as the registration and the frozen predicate as the instrument. It moved. As of this writing the row reads 1/35: one session has declared consequential actions with stated confidence and reconciled the outcomes — the first since the instruction shipped. One session proves nothing about effect size, and we won't pretend otherwise; the honest statement is that the row left zero after the instruction arrived and not before. Fittingly, that first compliant session reads unknown in the exposure column — it ran on a client from before the receipts shipped. The measurement is honest about its own blind spots even when the story would read better without them. Where this goes, and what we won't let it do The rounds ahead are deliberately boring: low-compliance instructions become rewrite jobs drafted by the fleet's own agents, every draft goes through human approval — instruction text is a prompt-injection surface, so no exceptions — and approved variants are A/B validated across real sessions once volume supports it. The system never rewrites its own instructions autonomously, in any round. The models never change; the interface to them does, under measurement. The compliance table ships with Firekeep — the Autopilot tab computes it live from your own sessions, with your own numbers, which will not be ours. If you run agents with an instruction file and have never seen this table for your team, that was our situation too, three days ago. The numbers were worth knowing. Explore Firekeep Install it AGENTS.md vs memory