They’re right that it’s the load-bearing objection. If the only “big model” you ever test is qwen3-max, “a society beats a big model” could just mean “our society beats one particular family’s flagship.” The honest way to find out is to run the claim against the actual frontier — every family you can reach — and see what survives. So that’s what I did. This entry is that test, and the small correction it forced on how I say the result.

Getting to the other families

Aftershock talks to models through a single provider chokepoint that was wired only for Alibaba’s DashScope (Qwen). The fix was small and additive: point the whole stack at any OpenAI-compatible endpoint via one environment variable, and route through OpenRouter, which serves every family behind one API. The one subtlety worth naming — because it’s exactly the kind of thing that silently corrupts a benchmark — is that a couple of request fields we send are Qwen-specific (enable_thinking, and a self-hosted reasoning toggle). Send those to a strict router in front of GPT-5 and it can reject the call. So the new path keeps those fields off the non-Qwen hosts, and the DashScope request body stays byte-identical to what it was. I also smoke-tested every model against the JSON decision contract before spending a cent, and made the cost ledger read real per-model prices so lives-per-dollar stays honest across families.

Then the experiment: the solo arm — one model runs the entire town, no protocol — on twelve models from ten families, ten paired seeds each, against the same cheap all-flash Qwen society (six qwen3.5-flash workers + the auction + doctrine) that has anchored this whole log: 106.0 lives saved, $0.025 a run, 4272 lives per dollar.

What the twelve models did

The headline is the top block. No single model’s solo beats the cheap six-flash society on lives. The eight frontier-class models — four American, four Chinese — land at 102–109 lives, every paired sign test at p ≥ 0.29, which at ten seeds with a standard deviation near sixteen is indistinguishable from the society. And they pay for the tie: 3–14× the cost, 306–1595 lives per dollar against the society’s 4272.

The correction I owe

Here’s the part I have to be straight about, because the cross-family run doesn’t just confirm the old story — it corrects it. When the only big model I’d tested was qwen3-max, its solo sat down at the swarm’s anarchy floor (a finding from a few logs back). That made it easy — too easy — to say “coordination beats a big model,” and to let that sound like a claim about lives.

Cross-family, that phrasing is too strong. A genuine frontier solo — GPT-5, Gemini 3.1 Pro, Opus 4.8, Grok, and DeepSeek V4 and GLM behind them — does reach the coordination ceiling on the outcome. It ties the society on lives. So the honest version isn’t “coordination beats a big model.” It’s: a big-enough model’s solo matches the cheap coordinated society on the outcome, and the society’s win is on cost — up to ~14× on lives-per-dollar against the flagship frontiers (a slimmer margin against the cheaper ones) — and it holds, in direction, across ten families. That’s a narrower claim than I’d been implying, and it’s the true one.

And there’s one honest dent I won’t bury: DeepSeek V4 Flash ties the society on lives and beats it on cost — 103.4 lives at $0.006 a run, four times the society’s lives-per-dollar. A cheap enough, good enough single model can win the cost argument too. It’s one model out of twelve, and the society’s edge is robust across the whole board, but “coordination always wins on cost” would be an overclaim, so: it doesn’t, quite.

What didn’t tie

Below the frontier, the picture is exactly the capability story you’d hope a good ruler shows. Mistral Large lands 11 lives short (p=0.002), Qwen3-235B twenty-six short (p=0.002), and the Llama-3.1-8B floor collapses to 24.6 lives — losing every one of the ten seeds — echoing the 1.7B collapse I found self-hosting earlier. There’s a real competence threshold to operate the town at all, and it sits somewhere above 8B and below the frontier, consistently, across families. That’s the other half of the robustness result: the substrate cleanly separates models that can do the job from models that can’t, and it does it the same way regardless of who trained them.

The honest bounds

Three, as ever. The runs aren’t byte-reproducible — the LLM layer never was (the provider ignores our sampling seed), so this is independent-seed inference, and I pair every model against the society by seed to cancel the world draw. The prompts are Qwen-tuned — they were written and iterated against Qwen, so a non-Qwen model that scores a little lower is partly being judged on prompt fit, not raw capability; the mitigation is that all twelve parsed the JSON contract cleanly, and the frontier tie (the load-bearing result) doesn’t depend on it. And the prices drift — these are OpenRouter list prices on the day, so read lives-per-dollar as an order-of-magnitude, not a decimal. One model-choice note: the only Kimi 2.7 on the router was the code-specialized build, which is a worse fit for a decision task and ran ~10× slower — but it still tied within noise, so it doesn’t move the verdict.

The whole thing cost about $14.50. That number is itself a small argument for the finding: I stress-tested the central claim of the project against every frontier model I could reach — GPT-5 to DeepSeek V4 — for the price of a sandwich, because the honest outcome metric is cheap to measure and the society it’s being compared against runs on models that cost fractions of a cent.

So: the reader was right to push, and the claim is better for it. Six cheap coordinated models match any single frontier model’s outcome — at up to a tenth of the flagship’s cost — and that holds across ten families, not one. It’s not the flashier “the small society beats the big model.” It’s the true one, and now it’s the one I’ll say.

One more data point, for anyone keeping score across projects: a sibling research effort of mine, Redoubt Arena, ran the same kind of check on a completely different simulated world — an enterprise-resilience mission instead of a disaster response — sweeping 15 models against a deterministic recovery baseline. Same shape of result: zero model runs beat the baseline outright; strong models only tie it. Different domain, same finding, twice now: a well-designed non-LLM baseline is a harder bar to clear than it looks, and it’s worth measuring before assuming an LLM adds value.

Live demo: https://aftershock.redoubtlabs.dev · Code: https://github.com/bluntmachetti/aftershock (method + verdict in docs/FIELD-NOTES.md §28; data under bench/results/2026-07-01-panelA-solo/)

Question 1: does a bigger model decide better?

This is the question that gates real money. If a larger C-suite model makes materially better decisions, you rent GPUs. If it doesn’t, you don’t. So you want to answer it before the hardware PO, on cheap hosted inference.

Aftershock’s society runs six role agents — a commander plus medical, rescue, fire, infrastructure, comms. The default roster is mostly cheap qwen3.5-flash workers. I held everything byte-fixed — the seeded world, the auction, the doctrine — and swapped all six roles across three price tiers via a single flag (bench --role-model), ten paired seeds, sixty ticks each:

Lives are flat. Every pairwise paired delta is statistical noise: plus over flash is +1.5 (sign test p=0.754), max over flash is +1.0 (p=1.000), max over plus is −0.5 (p=0.453) — all dwarfed by a per-seed standard deviation of ~17. Conformance barely moves (0.872 → 0.890 → 0.879). Meanwhile cost rises 9.7× and lives-per-dollar collapses 9.6×.

The tell is in the per-seed rows. On seed 11 the three tiers saved 140 / 136 / 136 lives; on seed 127, 83 / 82 / 82; on seed 73, 126 / 131 / 130. The model you pick barely shifts the outcome — the world you draw sets it. (That matches what we measured back in Log 004: the world accounts for ~79% of the variance between runs. The model tier is rounding error against it.)

This isn’t “models don’t matter.” An earlier self-hosted size sweep (recorded in docs/FIELD-NOTES.md §22) had a too-small model — 1.7B — collapse to zero lives saved: there’s a capability floor, and below it model size is everything. The finding is that once your roster clears that floor, paying more buys nothing the outcome can see. And the cheap lever we already shipped — written doctrine, which lifts protocol conformance by a credible +0.125 (p=0.031) at zero extra cost — moves exactly what a 10× model spend couldn’t. The verdict is a clean KILL: no GPU capex is justified for this task; spend the budget on the harness, not the model.

Question 2: when does coordination actually matter?

The running theme of this whole log is a coordinated society of small models versus an uncoordinated swarm, scored last week as a price of anarchy — the efficiency a resource scramble leaves on the table. But I’d only ever measured it at one level of scarcity. The open question is whether the coordination advantage is a constant, or whether it depends on the friction. Theory has a strong prior here: with no congestion there’s no externality, so there should be nothing for a coordination mechanism to fix.

So I swept it. Same seeded world, same arms, varying only the resource abundance — a uniform pool ladder from 12 of each unit (abundance) down to 1 (famine) — running the coordinated arms (scripted central heuristic, society auction) against the uncoordinated swarm. Eight paired seeds per level. Efficiency = fraction of imperiled lives saved:

The coordination advantage is an inverted-U, gated entirely by friction — and the surprising end is the top of the table. At abundance, coordinating is net-harmful. With units lying around unused, the auction is pure overhead: the uncoordinated swarm just grabs what it needs and saves more lives (79.3% vs the society’s 73.7%), winning on all 8 seeds, p=0.008 — the single strongest result in the whole sweep. Coordination only starts paying once contention appears: it pulls ahead through moderate scarcity, peaking at pool=2 (+7.3 points, a 1.18× price of anarchy). Then at the famine floor everyone collapses to ~20% and the gap vanishes — when nothing can be saved, there’s nothing to arbitrate.

The diagnostics confirm the mechanism rather than just the outcome. Contested resource losses climb monotonically as pools tighten — 12 → 222 → 612 → 1424 — and priority inversions stay at exactly zero until the famine level (then 26), which is its own small corroboration of Log 004: the auction allocates soundly right up to the point where the problem becomes unwinnable.

The two costumes, same body

Put the two experiments side by side and they say one thing. In Question 1, across a 10× span of model price, scripted and society track each other — the hand-tuned heuristic and the LLM society are within noise. In Question 2, across the entire scarcity ladder, they track each other again: 73.9/73.7, 72.4/72.1, 69.2/68.0, 50.7/47.1, 23.6/20.1. Whether the deciders are heuristics or a six-model society, and whether those models are cheap or expensive, barely registers.

What registers is the coordination mechanism, and only when the world is contended enough to need it. The lever isn’t model quality and it isn’t even “LLM vs hand-tuned” — it’s structure under scarcity. Spend your effort there.

What stays suggestive, and what doesn’t

The honest bookkeeping, because over-claiming is the fastest way to lose a reader’s trust (Log 006 taught us that the hard way).

The abundance reversal is significant — society loses all eight seeds, p=0.008. But the coordination wins in the middle of the curve are suggestive, not significant: +6 to +7 points is real-looking, but at eight seeds the sign test sits at p=0.289. That’s the same caveat the raw society-vs-swarm edge has carried for three logs now, and a uniform pool ladder doesn’t dissolve it — it would take more seeds at the pool=2 sweet spot to firm it. And Question 1’s flat result is a null at n=10 with σ≈17, so it rules out a large model-tier effect on lives, not a tiny one — but a flat line across a 10× price span, with no tier ordering, is itself the finding.

Two clean, reproducible results, then, with their uncertainty stated: bigger models don’t decide better above the floor, and coordination is worth nothing — sometimes worse than nothing — until the resources get scarce. Both point the same way, and both came from asking a sister project’s question on a substrate honest enough to answer it.

Build the ruler first, as ever. Then let it tell you where the money isn’t worth spending.

Live demo: https://aftershock.redoubtlabs.dev · Code: https://github.com/bluntmachetti/aftershock (method + verdicts in docs/FIELD-NOTES.md §26–§27; data under bench/results/2026-06-30-*)

Related work

Checked before publishing (Log 006’s lesson). The game-theory framing under Question 2 is the same literature Log 009 drew on; included here for the friction-necessity argument specifically.

• Hardin, G. (1968). The Tragedy of the Commons. Science, 162(3859), 1243–1248. https://doi.org/10.1126/science.162.3859.1243 — no friction, no externality; the abundance row is the boundary case where the commons problem disappears.
• Rosenthal, R. W. (1973). A class of games possessing pure-strategy Nash equilibria. International Journal of Game Theory, 2, 65–67. https://doi.org/10.1007/BF01737559 — congestion games: the inefficiency is a function of the congestion.
• Koutsoupias, E. & Papadimitriou, C. (1999). Worst-Case Equilibria. STACS ‘99, LNCS 1563, 404–413. https://doi.org/10.1007/3-540-49116-3_38 — the price-of-anarchy ratio used as the y-axis above.
• Roughgarden, T. & Tardos, É. (2002). How Bad Is Selfish Routing? Journal of the ACM, 49(2), 236–259. https://doi.org/10.1145/506147.506153 — the canonical result that uncoordinated play wastes a bounded factor, which the famine floor echoes (the gap re-compresses).

It’s the best framing question I’ve gotten — and one I couldn’t resist, because I’m an economics and game-theory geek at heart (my first degree was in Economics). That’s the honest reason this entry runs longer and digs deeper into the theory than usual. Because it turns out Aftershock isn’t like a game-theory problem — it is one, almost on the nose. So I spent today building a ruler for it and reading the literature to make sure I was using the words correctly. This is what the ruler said, what it can’t say yet, and where it points next.

The game hiding under the simulation

Strip the disaster theme away and Aftershock is a common-pool resource game. Each tick, six role agents claim from finite shared pools — ambulances, rescue crews, fire engines, fuel — toward missions that have a severity, a deadline, and lives at risk. Every claim one agent makes is a unit another agent can’t use: a textbook negative externality. That is precisely the setting where game theory predicts the uncoordinated outcome is inefficient — the classic tragedy of the commons (Hardin, 1968) — and where structure can claw the efficiency back. Elinor Ostrom’s Nobel-winning work (Governing the Commons, 1990) is the whole counter-argument: communities don’t need privatization or a central state to avoid the tragedy; they need institutions — shared rules everyone follows. That is a startlingly exact description of what we call doctrine.

The formal object for “players sharing congested resources” is a congestion game (Rosenthal, 1973), and the standard way to score how much selfish play costs is the price of anarchy — the ratio between the social optimum and the worst equilibrium — introduced as the “coordination ratio” by Koutsoupias & Papadimitriou (1999) and named by Papadimitriou (2001). The canonical worked example is selfish routing, where Roughgarden & Tardos (2002) proved uncoordinated traffic wastes at most a 4/3 factor with linear latencies. Swap “drivers on roads” for “agents on ambulances” and you have our benchmark.

Our four arms line up on exactly the coordination axis:

The society’s auction is, literally, a piece of mechanism design — the field that runs from Vickrey (1961), Clarke (1971) and Groves (1973) (the VCG family of efficient, strategyproof allocation rules). And the doctrine is best read not as Nash but as a correlated equilibrium (Aumann, 1974): a shared signal that all agents condition on, which can beat what uncoordinated best-responses reach on their own.

What the ruler measured

Raw “lives saved” is unbounded and trajectory-dependent, so it’s a poor yardstick for efficiency. Instead I measured the fraction of imperiled lives saved, grounded in the sim’s own exact accounting: every life that becomes at-risk is eventually saved, lost, or still-open, so

total_at_risk = lives_saved + lives_lost + open_remaining     (an identity, not a model)
efficiency    = lives_saved / total_at_risk                   (in [0, 1])

That’s aftershock poa, computed over recorded runs — deterministic, no API spend. Here is the result:

The headline isn’t the top line — it’s the clustering. Both coordinated arms land at ~66–67%; both uncoordinated arms land at ~58%. The structure, not the specific agent, is what moves the number. And the sharpest data point in the whole project is hiding in that table: the expensive single big model (solo) sits at the swarm’s anarchy level. For this allocation problem, coordination beats raw model size — six cheap models with a protocol out-deliver one big model without one.

Pairing society against swarm directly, pooled to fifteen seeds:

So the swarm leaves roughly an eighth of the achievable lives on the table relative to the society — a real price of anarchy — but the pairwise gap carries the same statistical caveat as the raw-lives number two logs ago: the CI excludes zero, the sign test doesn’t clear significance. The efficiency reframe is more interpretable (it’s bounded, and it cleanly separates coordinated from uncoordinated); it does not manufacture significance, and I’m not going to pretend it does.

What I can’t claim (and why I’m saying so)

Three honesty bounds, because the fastest way to discredit a game-theory framing is to over-reach with it.

The agents are not equilibrium-solvers. They’re LLMs following prompts, not utility-maximizers computing best responses. So “the society reaches a Nash equilibrium” is a category error — nobody is solving a fixed point. The honest statement is that the game has an inefficient uncoordinated region and an efficient coordinated one, and we measure where each arm’s emergent play lands. (The reason this still works is one of my favorite results in the field: Roughgarden’s smoothness framework (2015) shows price-of-anarchy bounds extend automatically to no-regret learning and correlated play — i.e., they bite even for messy, non-equilibrium agents. That’s the bridge from the textbook to a swarm of language models.)

The denominator is a ceiling, not a tight optimum. Efficiency = 1.0 means every imperiled life saved. That’s a rigorous upper bound, but it is not the best achievable outcome under finite resources — the true optimum is a multi-tick scheduling problem over the deterministic world, and it’s intractable to compute exactly. So I report “fraction of saveable,” never “fraction of optimal.”

On the hardest scenario, the order flips. On the brutal real-data NYC Hurricane Ida pack, the scripted central heuristic saves 32.2% and the LLM society saves 8.9%. The coordination edge is a synthetic-benchmark finding; the society is not magic on a savage real scenario. That belongs in the same paragraph as the win, not in a footnote.

This is also why I keep one skeptical paper open on my desk: Cemri et al.’s Why Do Multi-Agent LLM Systems Fail? (2025) catalogs how often multi-agent LLM setups underperform — exactly the null hypothesis a “society beats X” claim has to survive. Ours survives it structurally and suggestively, not decisively.

The experiments that would firm it

A ruler is only useful if it tells you what to build next. Three follow-ups, in order of value:

1. A self-enforcement test. Right now we know the society’s allocation is efficient (it eliminates priority inversions — see Log 004). We don’t know if it’s stable: could any single role do better by deviating from the auction outcome? Replay each contested tick and check for a profitable unilateral deviation. If there is none, the coordinated allocation is incentive-compatible — a self-enforcing equilibrium, not merely an imposed rule. That’s the difference between “we told them to cooperate” and “cooperating is their best move.”

2. A central-planner oracle for a tighter optimum. The save-everyone ceiling is loose. A strong omniscient greedy planner run through the same deterministic engine would give an achievable social-optimum proxy, turning “society reaches 67% of saveable” into “society reaches X% of what a perfect coordinator achieves” — the real price-of-anarchy denominator.

3. Auction strategyproofness. Our auction allocates by urgency bids. Can an agent win contested units by overstating urgency? The VCG line (Vickrey/Clarke/Groves) is the theory of allocation rules where honesty is the dominant strategy. Testing whether doctrine conformance keeps bids truthful — or whether a strategyproof rule does it for free — is a clean mechanism-design question we already have the harness to answer.

Turning the lens on real incident response

Here’s the part that makes this more than a benchmark curiosity. Real incident response already runs on a coordination doctrine. The U.S. National Incident Management System (FEMA, 2017) and its Incident Command System exist because uncoordinated multi-agency response to a disaster is a price-of-anarchy disaster: duplicated effort, contested resources, no clear allocation. ICS is, in our vocabulary, the correlation device — a shared playbook that every responding unit conditions on. Aftershock’s “doctrine” is a toy of exactly that idea.

And the resource problem under it is a well-studied optimization target: emergency dispatch and ambulance routing/location have a large literature (see the reviews by Mukhopadhyay et al., 2022 and Tassone & Choudhury, 2020), and multi-agent disaster response has been a grand challenge since RoboCup Rescue (Kitano & Tadokoro, 2001), with agent-based simulation used to plan resource allocation for major incidents (Hawe et al., 2015).

So a concrete future iteration of the app: a mechanism-design dispatch layer. Picture an incident commander facing several simultaneous incidents competing for the same scarce units. Today that allocation is often made under load, ad hoc. A decision-support layer could run the same auction + doctrine Aftershock uses — a transparent priority mechanism over real incident demand — and, crucially, report the price of anarchy of the status-quo allocation versus the coordinated one: “this dispatch pattern is leaving an estimated N% of saveable response-time on the table.” That’s the honest framing the whole project is built on — real demand and latency, simulated outcomes — extended into a what-if tool. Not a replacement for human incident command (the literature, and common sense, say the human doctrine is the point); a ruler the commander can hold up against their own decisions.

The other half is the agents. Generative-agent societies (Park et al., 2023), multi-agent debate (Du et al., 2023), and LLM negotiation under mixed incentives (Abdelnabi et al., 2023) are all converging on the same question Aftershock asks in miniature — and the open problems in getting them to cooperate reliably are exactly the ones Dafoe et al. (2020) laid out for Cooperative AI. A dispatch layer is a high-stakes, legible place to study them: the externalities are real, the doctrine is real, and the price of anarchy is measurable.

The honest takeaway

Coordination has a value, and now we can put a number on it: on this benchmark the protocol-free swarm pays a ~1.11× price of anarchy (pooled over fifteen seeds; ~1.16× on the five-seed four-arm cross-section), and — the cleaner result — coordinated play (whether a hand-tuned heuristic or an LLM society) beats uncoordinated play (whether a swarm or one big model) by ~8 points of saveable lives. The society-vs-swarm edge itself remains suggestive, p=0.118; the order flips on the hardest real scenario; and the agents aren’t really solving for equilibrium at all. The interesting work isn’t claiming a bigger win — it’s making the coordination self-enforcing, measuring it against a tight optimum, and pointing the same ruler at a real dispatch board.

Build the ruler first, as ever. Then let it tell you what to build.

Live demo: https://aftershock.redoubtlabs.dev · Code: https://github.com/bluntmachetti/aftershock (method + verdict in docs/EVIDENCE.md §3 and docs/FIELD-NOTES.md §25)

Related work

Every reference below was independently checked before publishing (Log 006’s lesson). Foundations of game theory and mechanism design; recent multi-agent-LLM work; and the incident-response literature the application section draws on.

Price of anarchy & efficiency of equilibria

• Koutsoupias, E. & Papadimitriou, C. (1999). Worst-Case Equilibria. STACS ‘99, LNCS 1563, 404–413. https://doi.org/10.1007/3-540-49116-3_38
• Papadimitriou, C. H. (2001). Algorithms, Games, and the Internet. STOC ‘01, 749–753. https://doi.org/10.1145/380752.380883
• Roughgarden, T. & Tardos, É. (2002). How Bad Is Selfish Routing? Journal of the ACM, 49(2), 236–259. https://doi.org/10.1145/506147.506153
• Roughgarden, T. (2015). Intrinsic Robustness of the Price of Anarchy. Journal of the ACM, 62(5). https://theory.stanford.edu/~tim/papers/robust.pdf
• Nisan, N., Roughgarden, T., Tardos, É. & Vazirani, V. (eds.) (2007). Algorithmic Game Theory. Cambridge University Press. https://doi.org/10.1017/CBO9780511800481

Congestion games & the commons

• Rosenthal, R. W. (1973). A class of games possessing pure-strategy Nash equilibria. International Journal of Game Theory, 2, 65–67. https://doi.org/10.1007/BF01737559
• Hardin, G. (1968). The Tragedy of the Commons. Science, 162(3859), 1243–1248. https://doi.org/10.1126/science.162.3859.1243
• Ostrom, E. (1990). Governing the Commons. Cambridge University Press. https://doi.org/10.1017/CBO9780511807763

Mechanism design & correlated equilibrium

• Vickrey, W. (1961). Counterspeculation, Auctions, and Competitive Sealed Tenders. Journal of Finance, 16(1), 8–37. https://doi.org/10.1111/j.1540-6261.1961.tb02789.x
• Clarke, E. H. (1971). Multipart Pricing of Public Goods. Public Choice, 11, 17–33. https://doi.org/10.1007/BF01726210
• Groves, T. (1973). Incentives in Teams. Econometrica, 41(4), 617–631. https://www.jstor.org/stable/1914085
• Aumann, R. J. (1974). Subjectivity and Correlation in Randomized Strategies. Journal of Mathematical Economics, 1(1), 67–96. https://doi.org/10.1016/0304-4068(74)90037-8

Multi-agent LLM systems

• Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. UIST ‘23; arXiv:2304.03442. https://arxiv.org/abs/2304.03442
• Du, Y., et al. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325. https://arxiv.org/abs/2305.14325
• Abdelnabi, S., et al. (2023). Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation. arXiv:2309.17234. https://arxiv.org/abs/2309.17234
• Cemri, M., et al. (2025). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657. https://arxiv.org/abs/2503.13657
• Dafoe, A., et al. (2020). Open Problems in Cooperative AI. arXiv:2012.08630. https://arxiv.org/abs/2012.08630

Incident response & emergency resource allocation

• FEMA (2017). National Incident Management System, 3rd ed. U.S. Dept. of Homeland Security. https://www.fema.gov/sites/default/files/2020-07/fema_nims_doctrine-2017.pdf
• Kitano, H. & Tadokoro, S. (2001). RoboCup Rescue: A Grand Challenge for Multiagent and Intelligent Systems. AI Magazine, 22(1), 39–52. https://doi.org/10.1609/aimag.v22i1.1542
• Mukhopadhyay, A., et al. (2022). A Review of Incident Prediction, Resource Allocation, and Dispatch Models for Emergency Management. Accident Analysis & Prevention, 165, 106501; arXiv:2006.04200. https://arxiv.org/abs/2006.04200
• Tassone, J. & Choudhury, S. (2020). A Comprehensive Survey on the Ambulance Routing and Location Problems. arXiv:2001.05288. https://arxiv.org/abs/2001.05288
• Hawe, G. I., et al. (2015). Agent-based simulation of emergency response to plan the allocation of resources for a hypothetical two-site major incident. Engineering Applications of Artificial Intelligence, 46, 336–345. https://doi.org/10.1016/j.engappai.2015.06.023

Honest. Also soft. A judge who knows statistics will read “+8.9, p = 0.118” and — correctly — discount it. Which forced the question I’d been able to dodge while the lives number still looked big: if your proudest result is only suggestive, what do you actually put on the front page?

The rule we landed on is the one that’s run through this whole project: lead with what survives scrutiny. Not the flashiest number — the one that’s still true after a skeptic checks it. Two of ours are. So I rewrote the README, the evidence pack, the submission, and the blog’s own headline around them.

What survives, #1 — written doctrine lifts conformance, credibly

Aftershock scores every agent against a two-tier playbook (a role envelope + a decision registry): how often does each role actually follow the protocol it’s given? That rate is team_alignment. The causal test is a paired ablation — same world seeds, same tools, the doctrine layer toggled off vs on — and it’s the cleanest result in the project:

The effect is positive on all 11 seeds across both runs, and at n=6 it finally clears the sign-test floor (five-for-five is only p=0.0625 — as strong as five coin flips get; the sixth seed tips it to p=0.03125). Crucially, doctrine buys this alignment at no lives cost — the lives delta in the same ablation is noise (p=0.375) in both runs. The structure makes the agents follow the protocol, not save more people; we don’t conflate the two.

And on the run a judge actually lands on — seed91-society, 65 ticks on real NYC Hurricane Ida dispatch demand — the society follows its doctrine 95.2% of the time (team_alignment = 0.9517). That’s the number I’d defend in a room full of Qwen engineers: structured instruction-following, measured deterministically, reproducible from the committed conformance file.

Conformance ≠ outcome. 95% alignment means the agents obeyed the protocol — it does not mean they rescued everyone. Outcomes (lives saved/lost) are a fully simulated model; we never claim the agents beat real outcomes. The conformance number proves instruction-following, full stop.

What survives, #2 — the cost-efficiency is real

The other claim a skeptic can’t knock down is about money. On the 4-arm benchmark (paired seeds, 60 ticks), six cheap qwen3.5-flash workers plus one qwen3.5-plus commander:

The society matches hand-tuned expert heuristics on lives (108.4 vs 106.8 — a coordinated team of small models holding its own against domain expertise) and out-delivers the single big model (108.4 vs 95.6) at 31% lower cost — which is ~65% better lives-per-dollar (3,069 vs 1,855). That’s the Qwen-track thesis stated in a way that doesn’t depend on a fragile p-value: cheap models, coordinated by a protocol, beat one expensive model for less money. (A later contract trim — Log 005 — cut another ~14% off the society’s per-run cost without touching conformance, on top of this.)

What we’re not leading with anymore

The +8.9-lives society-vs-swarm edge doesn’t get deleted — that would be its own kind of dishonesty. It’s directionally consistent (society wins 11/15), its CI excludes zero, and it’s the honest residue of the number we used to headline. It stays in the evidence pack, labeled suggestive:

By the harness’s own three-tier rule (credible = CI-excludes-0 and sign-significant; suggestive = exactly one; noise = neither), exactly one condition holds — so it’s suggestive, and that’s what we write. The point of the rule is that it demotes us, not just other people’s claims.

The evidence is finally visible on landing

There’s a postscript that belongs in a build log. While reframing the writing, I checked what a judge actually sees at https://aftershock.redoubtlabs.dev — and it was greeting visitors with “No runs found” everywhere and a Bench tab stuck on “Loading…” forever. All that carefully-firmed evidence was invisible on first contact.

The cause was almost funny: the public demo runs a back-to-back ambient simulation to keep the Live tab alive, and over thirteen days it had quietly written 32,709 throwaway run directories. The run-list endpoint read every one on every request; at 32k it took 62 seconds to answer and blocked the bench endpoint behind it. The fix was three parts — prune the firehose and cap it so it self-heals, list every curated run but only the newest handful of ambient ones, and bundle the demo arc into the repo itself so a fresh clone or a fresh box shows real data immediately instead of an empty panel. /api/runs went from 62s to under a second; the observatory now loads seed91-society — that 95%-conformance NYC-Ida run — on landing, no clicks.

It’s the same lesson as Log 006, wearing yet another hat: a system that intends to show its evidence isn’t showing it until you look at what a stranger actually sees. The blog could describe our credibility all it wanted; the app needed to put it on screen.

The honest headline

“Society achieves 0.95 doctrine conformance — credible at p=0.031 — at ~65% better lives-per-dollar than one big model” is less of a flex than “+28 lives saved.” It’s also the version that’s still true after you check it. Every figure here traces to a file in the repo (docs/EVIDENCE.md), and the field log keeps the receipts — including the ghost we chased and the headline we had to walk back. If there’s one thing this project is for, it’s that: pick the claim that survives scrutiny, and let the ruler demote you when it should.

Live demo: https://aftershock.redoubtlabs.dev · Code: https://github.com/bluntmachetti/aftershock

My Alibaba Cloud hackathon credits landed this week. The obvious move was to spend them on new experiments. I spent the first of them keeping that promise instead — and the number didn’t survive.

Build the audit trail before you generate the data

I had credits and a backlog. The temptation is to start running. But I’d already learned (Log 004, Log 006) that the expensive mistakes in this project aren’t bad runs — they’re believing runs I shouldn’t. So before spending a token, I built the thing that makes belief cheap and honest: an experiment tracker.

Not a big one. A sibling project of mine has a proper one — a database, schema-pinned ledgers, a hash-chained research log, viability certificates. I read all of it and then deliberately didn’t port it. Aftershock is a two-week hackathon repo with one model family; that machinery is paper-grade over-build here. What it actually lacked was small and specific: none of the ~dozen result folders carried the commit they were run on, the JSON had quietly drifted across versions, and there was no way to ask “which of my experiments are credible?” across the whole set.

So the tracker is three thin things: a provenance stamp on every result file (schema version, git SHA, which endpoint — cloud DashScope vs a local model — and a digest of the deterministic scripted baseline so a silent regression in the control is detectable); a one-file index built by a read-only script that walks every run and emits one queryable row each; and a link from a result to the field note it backs. About 350 lines, pure standard library, no new dependencies. The point of building it first is the same as building a ruler before you cut: you want the measurement in place before the thing you’re measuring exists.

The tool’s first job was catching itself

The first real experiment I pointed it at was a cheap, important one. An earlier finding (Log 002 territory) showed that written doctrine — a two-tier playbook the agents are scored against — raises how often the society follows its own protocol. At five seeds that effect was real on every seed but sat just shy of significance (a sign-test floor of p=0.0625, because five-for-five is as strong as five coin flips get). One more seed would tip it. So I ran a sixth.

The conformance lifted, six-for-six positive, sign test now p=0.031 — credible. I went to read it back out of the new index, and the index said the experiment’s verdict was: noise.

It wasn’t wrong about a number. It was reading the wrong number. The tracker computed its one headline verdict from the lives delta — which for a doctrine change is pure noise, by design, because doctrine buys conformance, not lives. The credible result was sitting right there in the data and the tool I’d built to keep me honest was filing it under “nothing to see.” The honesty layer was mislabeling on day one.

That’s the recurring lesson of this whole project wearing a new hat: a dashboard that intends to be honest isn’t honest until you audit what it actually surfaces. I taught the tracker to compute a separate conformance verdict with the same statistics, and then the doctrine result read credible. The fix paid off twice in one afternoon — because the very next experiment was the one I’d been avoiding.

The headline, firmed: +28 → a question mark

The promise from Log 004 was ~25 seeds. I went and got them. The society-vs-swarm contrast, re-run on the current code, across three batches, pooled into one paired test:

Read top to bottom, that’s a number deflating in real time. The same five seeds that gave +27.6 in June gave +14.6 on today’s code. Ten brand-new seeds came back at +6.0. Pool all fifteen and you get +8.9 lives, with a confidence interval [+2.3, +15.4] that just clears zero but a sign test (p=0.118) that doesn’t clear significance. The tracker calls that suggestive — exactly one of its two bars met, not both.

The culprit is the one Log 004 already named: a single seed. In the original five, seed 11 was a +88-life blowout that dragged the mean up; the other four averaged about +12. Once the sample is big enough that no one lucky draw can carry it, the magnitude regresses to a small, not-significant edge. The direction still holds — the society wins 11 of 15 seeds, it does beat the uncoordinated swarm more often than not — but the number on the homepage does not survive being firmed.

So I did the unglamorous thing. I rewrote the README, the submission writeup, and the judge-facing evidence pack to lead with what’s true: society-vs-swarm is suggestive, not statistically significant (+8.9 at n=15, CI [+2.3, +15.4] excludes 0 but the sign test p=0.118 doesn’t clear significance), and the earlier +28 was a small-sample overestimate. The blog posts that reported +28 at the time stay exactly as they were — they were honest about what we knew then, and rewriting history would be its own dishonesty. This is the next chapter, not a retcon.

And the cost trim, kept honest

While the tracker was on, I closed one more open question. A while back I trimmed the static prompt every agent re-sends each tick and cut run cost ~14% in a clean A/B ($0.0411→$0.0353, Log 005) — about ~16% cumulatively since the 2026-06-11 launch ($0.0423→$0.0353), a different baseline. I’d flagged a small conformance dip from that trim as a “watch-item — could be noise.” Now that conformance is the part of the story that survived, I owed it a real answer, so I built a clean A/B toggle and ran it at ten seeds.

The dip is real but tiny: −0.019 team-alignment, with a confidence interval that just clears zero but a sign test that doesn’t (the tracker calls it suggestive) — and zero lives cost. So the −14% cost cut is nearly free, not perfectly free. Worth knowing, not worth reversing. The point is that I now know which it is, instead of guessing.

What survives

Here’s the honest scoreboard after a session spent measuring instead of adding:

• Cost-efficiency — robust. Six qwen3.5-flash workers under a qwen3.5-plus commander match well-tuned expert heuristics (108 vs 107 lives) and a single big model (108 vs 96 for solo qwen3-max) at ~$0.035 a run — over 50% better lives-per-dollar than the big model. That’s the claim I’d stake the submission on.
• Conformance — now credible. Written doctrine lifts protocol-following by a verdict that finally clears significance (p=0.031, six-for-six).
• Society vs swarm on lives — a question mark. Suggestive (11/15, CI [+2.3, +15.4] excludes 0) but not significant (sign test p=0.118). The marquee magnitude was the weakest claim, and it’s the one that didn’t hold.

Losing your headline number is supposed to feel bad. It mostly feels like relief, because the alternative was a judge running aftershock bench and finding +9 where the slide said +28. The tracker earned its entire existence in one session by demoting me, on purpose, before anyone else could.

If there’s a takeaway sharper than Log 004’s “build the ruler first,” it’s this: firm your proudest number first. The figure you most want to be true is the one most likely to be a small-sample flatter, and it’s the one with the most riding on being wrong. Point the audit trail at it on day one. The best day a measurement tool has is the day it tells you the thing you were about to brag about isn’t there.

Try it live: https://aftershock.redoubtlabs.dev · Read the code: https://github.com/bluntmachetti/aftershock (the firming lives in aftershock ablation; every number above traces to a file in the judge evidence pack)

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.

With the hackathon deadline close, the temptation was to bolt on a flashy feature. The measurement from the earlier logs said otherwise. Aftershock was already feature-complete; what it lacked wasn’t capability, it was legibility — a judge couldn’t quickly check that the numbers were real. So the final push wasn’t features. It was proof: a Decision Receipt that chains any contested ruling (kernel decision → the agent’s own stated rationale → cost → recorded outcome), confidence intervals and a sign test on the benchmark, and the capstone — a one-page Evidence Pack whose entire promise is: every number here traces to a file in the repo. Go check.

It’s the judge’s dream fact-check target. So before I shipped it, I did the judge’s job. I ran a multi-agent adversarial pass that re-derived every figure in the pack from its source — the results.json, the run manifests, the scenario JSON — and asked one question per number: does this actually trace?

The honest headline: 42 of 47 figures matched exactly. The mean lives (society 103.2, swarm 75.6), the costs, the lives-per-dollar, the NYC-Ida demand and latency baseline (948s, 16.5% held) — all clean. But the five that didn’t trace are the whole point of this post, because the document that failed them was the one I wrote specifically so it couldn’t.

The artifact that dares you to check it

The Evidence Pack opens with a line I was proud of: “Every number below traces to a file in this repo — no figure is asserted without a source path.” Each claim sits next to its source: a results file, a run.json field, a path inside scenario.json. That framing is a commitment. It also means a single wrong source citation doesn’t just look sloppy — it detonates the whole document, because a judge who runs jq on the cited field and gets back null will stop trusting every other number on the page.

So that’s exactly what the audit checked: not “is the value plausible,” but “if a skeptic runs the command next to this number, do they get this number back?”

So I checked it — and three citations lied

None of these flattered the project — the wrong p-value actually understated how cleanly the swarm loses (0.0625 is the strongest verdict five seeds can give; 0.375 was a number that wandered in from the adjacent row). The pack_digest value was even correct — it just pointed at the wrong place to find it. But that’s the trap with a proof artifact: honest intent plus a wrong pointer reads exactly like a lie to someone checking. A judge running jq .pack_digest scenario.json, getting null, and concluding the whole pack is hand-waved would be right to. Every one of these got a corrected source — the digest now cites the code that computes it and the run.json field that stores it; the p-value reads 0.0625.

A small, sharp aside: the AI reviewers I’d used to pressure-test the plan made the same class of error. One confidently recommended I “ship the scenario pipeline” — a thing already shipped weeks earlier (caught by checking the disk). Another quoted a flattering conformance figure of 0.915; the real number is 0.759 — it had been unable to read the gitignored file and filled the gap with something plausible. Models hallucinate sources too. The defense is identical: trace it.

The one that stung: a flagship at 95% conformance that saved 8 of 90 lives

The traceability misses were fixable typos. This one was a judgment failure, and it’s the reason the post exists.

The pack showcased seed91-society — the society running the real NYC Hurricane Ida scenario — as the flagship demo, headlined by its doctrine conformance of 0.95: “Qwen follows the structured doctrine 95% of the time over a full scenario.” True number. Real signal. And next to it, the pack said nothing about how the run actually went.

Conformance measures whether agents follow their playbook. It says nothing about whether they win. Under Ida’s real overwhelming demand, this run followed doctrine almost perfectly and still lost 82 of 90 lives. Both facts are true and they don’t conflict — but putting “95% disciplined” in the spotlight and leaving “saved 8 of 90” off the page implies success the run never had. A sharp judge who opened the run.json would find 8/82 and, fairly, distrust the rest.

This is the cherry-pick failure mode, and it’s seductive precisely because every individual number is real. You don’t have to fabricate anything to mislead — you just have to be selective about which true number gets the headline. The fix wasn’t to bury the run; it was to state it whole: a caveat now sits directly beside it — “conformance measures instruction-following, not lives; this run scored 0.95 but saved only 8 of 90 under heavy Ida demand.” That sentence makes the pack stronger, not weaker, because it’s the sentence a skeptic was going to write for me.

It wasn’t only the pack

Once I started auditing, the pattern repeated across every honesty surface I’d built that week:

• The Decision Receipt labeled the commander’s rulings “decided by kernel.” The commander is an LLM arbiter, not the deterministic engine — calling its calls “kernel” decisions is exactly the kind of authority-laundering the receipt exists to prevent.
• It showed a ruling as GRANTED when the auction had accepted the bid but the kernel then rejected the dispatch downstream — implying a resource reached a mission it never reached.
• It printed a town-wide lives-saved delta inside a card scoped to a single ruling, so opening the receipt for a declined call could show it next to “+1 saved.”

Three honesty bugs in the component whose entire job is honesty. All caught the same way: an independent pass that didn’t trust the summary and went to the source. All fixed before any of it reached a judge.

The takeaway

Aftershock’s thesis, from Log 001, has been one sentence: don’t trust the summary — check the receipts. It’s the argument for the whole project: typed decisions over chat transcripts, paired seeds over a single triumphant run, provenance labels over a confident demo.

This week that argument turned around and pointed at me. I built the most honest artifact I could — a pack that invites fact-checking — and it still shipped a wrong p-value, a citation to a field that doesn’t exist, and a flagship that omitted its own outcome. Not from bad faith. From the ordinary gravity that pulls every summary toward looking better than the thing it summarizes.

The lesson is narrow and, I think, the most useful one of the whole build:

Intending to be honest is not the same as being honest. A proof artifact is itself a claim, and claims need an adversary. The Evidence Pack only became trustworthy after something tried to break it — and the thing that broke it wasn’t more good intentions, it was the same boring discipline as the ruler in Log 004: re-derive the number from the source, and believe only what survives.

If you build a system to keep agents honest, point it at yourself last. That’s where it finds the most.

Try it live: https://aftershock.redoubtlabs.dev · Read the code: https://github.com/bluntmachetti/aftershock (the corrected proof bundle is docs/EVIDENCE.md; the receipt is web/src/components/DecisionReceipt.tsx)

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.

I expected the ruler to catch noise — a win that’s really a coin flip. It did that. But it also kept catching something subtler and, honestly, more embarrassing: levers that would work — move the exact number I aimed them at — and change nothing that matters. I came within one code edit of shipping a “fix” whose entire effect would have been to make a metric look better while the system behaved identically.

This is a post about outcome-neutral metrics, and the discipline of asking, before you optimize a number, whether the number does anything.

The setup: the society’s real story is cost and discipline, not lives

A quick recap of where the measurement left me. On Aftershock’s task, well-tuned scripted bots match every LLM arm on lives saved — the negotiation protocol, not model IQ, carries the result. And under genuine triage there’s no detectable lives edge of the society over an uncoordinated swarm. So the society’s honest, defensible value isn’t more lives — it’s cost-efficiency (small models, coordinated, at a fraction of a big model’s price) and conformance (a deterministic measure of whether agents actually follow their doctrine).

That reframing matters for everything below: it means a “win” on lives is usually noise, and a win worth shipping is one that moves cost or conformance — without quietly costing the other.

Lever 1 — does discipline cost lives? (an old scare, resolved)

Early on I’d given every agent a two-tier playbook — a “doctrine” — and a checker that scores how well they follow it. Writing the doctrine clearly raised conformance. But a single-seed run had a worrying side effect: the doctrine run saved fewer lives (96 vs 113). Discipline isn’t free — every instruction you add to a prompt is more to read and obey. I’d left it flagged as outcomes TBD.

n=1 is not a result, so I built a same-arm ablation: run the society with and without doctrine on the same five seeds, and — because conformance is the deterministic, low-variance signal — make the report lead with conformance and treat lives as the noisy secondary.

The scare was a small-sample artifact. Doctrine raises conformance on every seed, and lives are flat-to-slightly-up — the n=1 “−17 lives” was noise. Discipline here is not paid for in lives. That clears doctrine as a real, free conformance lever and tells me where the rest of the session should aim: at the one role that still won’t follow it.

Lever 2 — the agent that won’t follow the rules, and the guard I almost built to cheat

That role is the infrastructure agent. Even with doctrine, it’s the one specialist that stays below 0.70 conformance. The diagnostic showed why — three distinct failures, not one:

• Urgency inflation: it marks every request urgency 9, even low-severity ones with a distant deadline.
• Resubmitting rejected work: it re-issues a decision the engine just rejected.
• Impossible repairs: it calls repair_road on districts that aren’t blocked, or when there’s no repair crew free.

First I did the obvious thing: rewrote its prompt to tie each rule to the exact fields it already sees in its observation. That cleanly fixed urgency (0.35 → 1.00 conformance on that rule) — a numeric threshold the model can apply. But the impossible-repairs rule barely moved. Telling the flash model “only repair a district on the BLOCKED line, and only if crew ≥ 1” just… didn’t take. Precondition gating is harder to prompt than scalar calibration.

So I reached for the backlog idea: a deterministic guard — enforce the precondition in code so an invalid repair can never happen. And then, tracing the code to place it, I stopped.

The engine already rejects those repairs. An invalid repair_road is validated and declined before it consumes anything — zero resources, no effect on the world. And the conformance metric counts the agent’s attempt. So a guard that intercepted the attempt would do exactly one thing: make the conformance number go up by hiding the agent’s behavior from the record. The world would play out identically — same lives, same cost — and the scoreboard would look better.

That is the cleanest example I’ve hit of an outcome-neutral metric: a number you can move without changing anything real. Building the guard would have been reward-hacking my own benchmark. I didn’t build it. The honest version of the question isn’t “how do I make the number go up” — it’s “is the behavior actually fixable, and does fixing it matter?”

So I tested that instead: I gave just the infra agent a stronger model (qwen3.5-plus instead of flash) and measured, changing nothing else.

So the stickiness was a model-capability floor, not a prompt bug: the bigger model gates the preconditions the smaller one ignores. A real finding — but look at the trade. It buys near-perfect conformance on an outcome-neutral rule (the bad repairs cost no lives to begin with) for +33% cost. Paying a third more to perfect a metric that doesn’t move lives, on an arm whose whole pitch is cost-efficiency, is the wrong default.

So it ships as an opt-in operating mode — one flag, --role-model infrastructure=qwen3.5-plus — not the published default. Flip it when discipline matters more than dollars; leave it off for the lives-per-dollar story. The finding (it’s a capability floor) is the durable result, independent of which default you choose.

Lever 3 — the one that actually paid: stop re-sending the prompt

With lives ruled out and conformance either free (doctrine) or expensive (infra model), I turned to the axis the society can actually win on: cost. And here the harness did its best work as a profiler before it did anything as a judge.

Where does the money go? The commander sends 31,900 prompt tokens per run but only 2,300 of completion. Decomposing one call: a 941-token system prompt re-sent on every single tick, plus only ~160 tokens of actual world observation. Across all six agents, ~85% of a run’s prompt tokens are the same static prefixes, re-sent every tick — roughly 60% of total cost is the model re-reading instructions it already read last tick.

The obvious fix is caching — most providers bill a repeated prefix at a discount. So I probed it: a ~1,600-token stable prefix, sent three times in a row, on both Qwen models. The response told me plainly:

prompt_tokens_details: { text_tokens: 1589 } — no cached_tokens field, no discount, full price every call.

DashScope’s international compatible-mode endpoint doesn’t cache our prompts. That’s a clean negative result worth knowing: it means our cost ledger is accurate, not pessimistic — there’s no free accounting win hiding in cache hits. The only way to cut a re-sent prompt is to make it shorter.

Doctrine is off-limits (it’s the conformance lever from Lever 1) and the role instructions are behavioral, so the target was the output contract — the JSON-format boilerplate every agent carries. I compacted the multi-line schema to one line, deduplicated rules that were stated twice, and tightened the proposal descriptions — keeping every capability and field name, just terser. Commander prefix: 941 → 835 tokens. Then the paired A/B, against an identical baseline where only the contract differs:

A real, largely deterministic 14% cost cut — the token reduction is fixed; only the small completion varies — for a 21% gain in lives-per-dollar, the headline efficiency number. Lives even drifted up. The one thing to respect: conformance ticked down 0.037, but the sign test says that’s not a credible effect (p=0.375), and it sits inside the run-to-run band conformance has always wandered in (0.85–0.92). By the same bar I used to reject the +16-life ghost last post, I can’t claim this conformance dip either — so I kept the trim and logged the dip as a watch-item, not a regression.

The deeper lesson hid in the disappointment: the pure-redundancy trim was tiny (~16 tokens). Most of that 941-token prefix is irreducible — doctrine the agents need, decision vocabulary they act on, a schema that keeps the JSON valid. The society’s ~$0.04/run is mostly structural. −14% is close to the safe ceiling without touching doctrine or model tier. Worth banking; not a bottomless well.

The pattern: four levers, one bar

Before chasing each lever I made a workflow of read-only agents scope it against one question — does moving this metric change a real outcome? It skipped four backlog ideas outright (a deadline-sort for a failure mode that’s already near zero; reducing “redundant” bids the auction ignores for free; a temperature sweep premised on a cost number that turned out wrong). Each was an outcome-neutral mirage, the I1-guard pattern again: a number you can move that moves nothing else.

What survived the bar is exactly the society’s honest story:

1. Doctrine raises conformance on every seed, free of lives cost.
2. Conformance has a price ceiling — the last stubborn role needs a bigger model, which costs more than the (outcome-neutral) metric is worth as a default, so it’s a switch.
3. Cost is real and movable — −14% by not re-sending what the model already read, the cap set by how much of the prompt is genuinely load-bearing.

No grand lives breakthrough. But three claims I can defend in front of a judge, and a fistful of features I didn’t ship because the measurement said they’d only flatter the scoreboard.

The takeaway

Last post’s lesson was a single measurement is a coin flip wearing a lab coat — pair it, test it, know when to stop. This session added the sharper one:

Before you optimize a metric, prove the metric does something. The most seductive failure isn’t a noisy win — it’s a clean win on a number that’s decoupled from reality. I could have made my conformance score jump by intercepting decisions the engine already discards; it would have demoed beautifully and meant nothing. The guard against that isn’t more statistics. It’s tracing the metric back to an outcome — lives, dollars, missions — and refusing to celebrate a move that doesn’t reach one.

Make it measurable. Believe only what survives the measurement. And check that the thing you’re measuring is connected to the thing you actually care about.

Try it live: https://aftershock.redoubtlabs.dev · Read the code: https://github.com/bluntmachetti/aftershock (the levers are in docs/FIELD-NOTES.md §18–21; the cost trim is --role-model / the output contract)

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.

I didn’t build it. I built a ruler first — the boring statistical plumbing that tells you whether a change actually did anything. And the ruler earned its keep three times over in one sitting: it killed the headliner before I wrote a line of it, it caught a +16-life win that wasn’t real, and it made me put a caveat on my own flagship number.

This is a post about the least glamorous part of the project — measurement — and why it turned out to be the most valuable.

Why a ruler, before any tuning

Aftershock’s engine is byte-deterministic: same seed, same disaster, same outcome. But the agents are not. The published society result — 103.2 ± 23.6 lives — has a ± for a reason, and the first thing the ruler did was tell me exactly what that ± is made of.

Two cheap experiments:

• Does Qwen Cloud honor a sampling seed? I threaded a deterministic per-call seed through every request and ran the same world twice. The two runs diverged on the first tick, exactly like two un-seeded runs. DashScope accepts seed but doesn’t make sampling reproducible — so the LLM arms are irreducibly stochastic, and that ± really is sampling noise, not measurement slop.
• How much of the noise is the model, and how much is the world? Running each seed three times and decomposing the variance: 79% of it is which disaster you drew (between-seed), only 21% is the model’s run-to-run wobble (within-seed).

That second number is the whole game. If most of your variance is the scenario, then comparing a mean against a mean is mostly comparing coin flips — unless you pair: run the control and the treatment on the same seeds and difference out the world. Concretely, to detect a +10-life effect at decent power you need ~10 paired seeds, versus ~45 unpaired. Same data, a quarter of the runs, because pairing deletes the 79%.

So the ruler is: run both arms on identical seeds, take the per-seed difference, and judge it with a sign test, a bootstrap confidence interval, and a power curve — aftershock ablation. Nothing fancy. Just enough to stop me believing a number I shouldn’t.

Episode 1 — the diagnostic killed the headliner

The “biggest lever” exists to fix a specific pathology: a priority inversion. Under all-or-nothing granting, a high-priority incident that needs three ambulances can lose the whole pool to a later, lower-priority incident that happens to fit the two that are left. Partial grants would fix that. Worth +10–15 lives — if it happens.

So before building the fix, I built a diagnostic that reads the recorded auction outcomes and counts how often it actually happens. The answer, across nine society runs:

Zero.

Then I made the world harder — I added a knob to tighten the resource pools — figuring scarcity would surely manufacture the contention. Still zero. I cranked it to the harshest setting the sim allows (every pool at 2, where a fifth of all incidents fail outright):

Across the default and tight worlds: 133 contested auction losses, every single one legitimate — the loser always had lower-or-equal priority than the winner. The auction’s arbitration is just correct: it serves the highest-priority incident first, and the agents don’t over-request in the way that would let a low-priority bid steal the remainder. Only at brutal scarcity do a handful of inversions appear, and even then they’re dwarfed 70-to-1 by plain shortage (the pool is simply empty — there’s nothing for a partial grant to split).

The verdict wrote itself: don’t build the lever. It optimizes a problem this system doesn’t have. That’s the harness paying for itself before it cost a thing — a feature un-built is a feature you don’t have to maintain, document, or quietly regret.

(A humbling footnote: the first version of that inversion detector was dead code — by the resolver’s ordering it could never actually fire — and an adversarial code review caught it. Even the ruler needed measuring. I rebuilt it to reconstruct winners-versus-losers from the recorded grants, and then trusted the zeros.)

Episode 2 — the +16 that wasn’t

Fine: the auction isn’t the bottleneck on the easy task. But the easy task is easy — ~96% of incidents get resolved, so there’s barely any room for any change to move lives. The honest next question: under genuine triage — a world where you can’t save everyone and the order you serve people in decides who lives — does the society’s coordination actually beat the uncoordinated swarm by more?

I dialed in that world (the harshest setting: lives saved ≈ lives lost) and ran a paired society-vs-swarm ablation. Five seeds. The result looked great:

Δ = +16.2 lives, 95% CI [+3.2, +29.6] — excludes zero. Society stable at 63–72 lives across every seed; swarm volatile (32–66), collapsing on the two hard draws (39 and 32 lives).

A clean story practically wrote itself: coordination buys graceful degradation under stress. The harness’s own auto-verdict even printed “credible improvement.” I wanted to believe it.

But the same readout had two warnings I’d built in precisely so I couldn’t ignore them: the sign test was non-significant (p=0.375) and the statistical power was 0.57. The confidence interval said yes; the more conservative tests said not yet. The honest move with a disagreement like that is not to pick the answer you like — it’s to add seeds. So I added six more (the harness resumes the runs it already has, so only the new ones cost anything) and re-ran at eleven.

The effect evaporated. Four of the six new seeds came back with the swarm beating the society. The entire +16 had been carried by two lucky draws where the swarm happened to faceplant; the “society is rock-stable” story was the same small-sample mirage (at eleven seeds the society’s range is 32–80, not the tidy 63–72 I’d seen at five). There is no detectable lives advantage here. The power curve says confirming even the residual +4.7 would take ~88 seeds — which is the harness’s polite way of saying stop.

This is the single most important thing the ruler did. A +16-life result with a confidence interval that excludes zero is exactly the kind of finding you screenshot, put in a slide, and ship. It was noise. I caught it because I’d committed in advance to a test I couldn’t argue with after the fact.

Episode 3 — turning the ruler on my own headline

If a +16 can be a ghost, what about the project’s flagship claim — that the coordination protocol is worth +28 lives? I owed it the same scrutiny, so I ran the already-published numbers through the same paired test. The result, to my relief, held up where the +16 didn’t — but not cleanly:

• Directionally solid: society ≥ swarm on all five seeds (four wins, one tie, zero losses). Unlike the harsh-world ghost that went both ways, this gap is real in sign. The qualitative claim — the auction helps — survives.
• Magnitude soft: the precise “+28” is leaned on hard by a single seed (+88 there; the other four average +12.5), the sign test is p=0.125, and the power is 0.42. To put a tight interval on the number you’d want ~25 seeds.

So I’m keeping the claim and adding a caveat I can defend: the coordination protocol reliably helps; the headline figure is an n=5 mean with a wide interval. That’s a weaker sentence than “+28 lives,” and it’s the true one.

What the ruler cost, and what it bought

The whole measurement campaign — 41 paired runs across every world — cost $1.41 in Qwen Cloud tokens, tallied from the run ledgers. For that, I got three results I’d have paid far more for in wasted effort and overconfidence:

1. Don’t build the “biggest lever.” It fixes a pathology that occurs <1% of the time.
2. There’s no lives edge under triage. The +16 was a false positive; the honest story for the society is cost-efficiency and discipline, not out-saving the swarm under scarcity.
3. Caveat the +28. Real in direction, soft in magnitude.

None of those are features. Two are flat “don’ts” and one is a hedge. And they’re the most valuable output of the session — because the alternative was building a feature against a problem that isn’t there, and believing a win that wasn’t real, in a slide deck, in front of judges.

The takeaway

In a stochastic system, a single measurement is a coin flip wearing a lab coat. The plumbing that protects you from it — pair to kill the variance you don’t care about, judge with a test you picked before you saw the answer, and let a power curve tell you when to stop — isn’t bureaucracy. It’s the difference between knowing and hoping. Build that ruler before you build the thing you’re itching to build, because the ruler’s best days are the ones where it tells you no.

And hold your own dashboard to the same bar: my ablation tool’s auto-verdict cheerfully called the +16 “credible,” and the sign test is the only reason I didn’t believe it. The tool that keeps you honest still needs you to read all of it.

Make it measurable. Then believe only the part that survives the measurement.

Try it live: https://aftershock.redoubtlabs.dev · Read the code: https://github.com/bluntmachetti/aftershock (the harness is aftershock ablation / aftershock diagnose; the findings are in docs/FIELD-NOTES.md §13–17)

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.

That sentence is the entire project. And until last week you could only read it — in a scrolling text feed of granted / pool exhausted rulings off to the side of the map. The thing that matters most was the thing you couldn’t see happen.

So this build was about making the mechanism legible: a Mission Control rebuild of the map that draws the auction itself, live, on the city. It’s also a small parable about how a picture can lie exactly as easily as a number — and on a project whose whole pitch is honesty, the picture doesn’t get a pass.

From “map” to mission control

The old map was honest but generic: district blocks, mission pins, a panic gauge. Fine. But it read like a dashboard, not an emergency operations center — and it buried the one event you’d want a commander watching for.

The rebuild is an EOC command view. A condition state up top (RED/AMBER/BLUE/GREEN, driven by the nearest deadline, worst severity, and public panic), a tile map of the city, and — the point of the whole exercise — a contention overlay.

When two incidents reach for the last unit of a resource in the same tick, the map draws it: a dashed line from the district that lost to the district that won, labeled with the resource (AMB CONTESTED, RPR CONTESTED), a halo on both incidents. You watch the auction arbitrate the Qwen agents’ requests in real time. The qwen3.5-plus commander and the kernel’s auction stop being a feed you skim and become a thing you see resolve, tick by tick.

And it’s derived from data that was already there. Every tick record carries the agents’ typed resource_request proposals and the auction’s rulings. The overlay is a pure function of those — no new engine state, no new field, nothing for the deterministic core to even notice. Same pack, same seed, byte-identical run, now with the contention drawn on top.

The part a review caught

Here’s where it gets honest. The first version of that overlay was subtly wrong, and it took an automated code review on the pull request to catch it.

The auction can grant the same resource to several incidents before the pool runs dry. Picture one tick, three incidents reaching for ambulances against a pool that can only cover two of them:

m3 lost. But to whom? My first cut linked every loser to the first winner it happened to process — here, m1 in Harbor. So the map drew Market’s “you lost this” arrow at Harbor, when the auction’s own ruling says plainly: pool exhausted: ambulance granted to m2 — m3 lost the last unit to m2, in Hospital. The picture pointed at the wrong district.

The reviewer flagged it as a misattribution, and it wasn’t theoretical — it was live in the demo data. The very first contested tick of our headline society run drew the contention line at the wrong district. A plausible-looking arrow, confidently wrong.

The fix was to stop guessing and read what the auction already said: parse the winner named in each loser’s own ruling, and link to that district. It also made the code simpler — the whole “figure out who the winner probably was” scaffolding just deleted.

I keep a rule for the benchmark: never claim a number you didn’t measure on identical worlds. This was the same rule, wearing a different hat. A visualization is an inference too, and it earns the same scrutiny as a number — more, even, because a clean line on a map feels like ground truth in a way a table never does. The map can be confidently, beautifully wrong. On a project that puts REAL / MAPPED / INFERRED / SYNTHETIC labels on every field precisely so nothing overclaims, an overlay that quietly fingers the wrong district is the same sin in a prettier font.

Keeping a reskin honest

It’s “just frontend,” but the failure modes of a frontend change on a project like this are specific, so it got the engine treatment:

• The color contract is frozen. Society stays cyan, the baselines stay amber — the same coding the side-by-side compare view leans on. The redesign added exactly one new color (a caution yellow for contention) and reused the existing signals for everything else, so no other view shifted a pixel.
• The honesty surfaces stayed first-class. The reality strip — real first-on-scene latency vs. the agents’ simulated response, the INFERRED badge on lives-at-risk, the never-fabricated-when- null rule — all survived untouched, including on the live NYC Hurricane Ida pack with its borough names and provenance.
• The blast radius was provable. I pulled the rich incident markers into a shared module so the compare view’s two synced maps render from the exact same code but never inherit the overlay — the one map that should change changed, and the one that shouldn’t was untouched by construction, not by hope.
• Determinism never moved. Frontend-only, so the aftershock verify digest check couldn’t regress — and it didn’t.

Then it went out the way everything does: local → staging → production, verified at each hop, live now at the public demo.

The takeaway

The benchmark exists to make the result falsifiable: here’s the baseline, here are the identical worlds, here’s the measured gain. This build was the same instinct aimed at the mechanism: make the coordination you can’t otherwise see legible enough to watch — and then hold the picture to the same bar as the numbers, including the part you only find when someone reviews it and asks “wait, who actually won?”

Make the mechanism visible. Then make sure the visible thing is telling the truth.

Try it live: https://aftershock.redoubtlabs.dev — load a society run and scrub to a contested tick · Read the code: https://github.com/bluntmachetti/aftershock

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.

Then I did the thing I keep telling everyone else to do, and which is the entire point of this project: I benchmarked it before believing in it.

The benchmark told me to turn it off.

This post is that result — because a negative result you measured honestly is worth more than a feature you shipped on faith.

What “native function calling” replaced

In JSON mode, each agent’s system prompt carries a compact prose contract — here are your actions, here are the fields, return JSON — and the model returns a JSON object the engine validates. In tool mode, that same action vocabulary becomes a set of OpenAI-style tools definitions sent on every request, and the model emits structured tool_calls instead. Same decisions, same auction, same negotiation protocol underneath. The only thing that changed is how the action space is described to the model and how the model hands its choices back.

I kept the change behind a flag (--society-tools) and a force_tools switch, so I could run the exact same five paired seeds both ways and let the numbers decide. Same disasters, byte for byte. Only the calling convention differs.

The numbers

Read that top to bottom. Tool calling held lives saved within the noise — 98.2 vs 103.2 sits comfortably inside one standard deviation (±23) — while cost roughly doubled and latency rose ~2.5×. The one place it moved measurably in the wrong direction was missions failed: 0.8 vs 0.4, twice as many lapsed deadlines, plausibly downstream of the higher latency (the qwen3.5-plus commander even hit a couple of request timeouts under the heavier payloads).

If you’re tempted to read the 5-life dip as a real regression, the paired-seed view talks you out of it — tool mode was lower on four seeds, higher on one:

That’s noise with a faint downward lean, not a signal. So the honest summary is brutal in its simplicity: native function calling cost twice as much to do the same job, slightly worse.

Why — and why no amount of prompting saves it

The cause isn’t a bad implementation or an untuned prompt. It’s structural, and it’s worth internalizing if you build multi-agent systems.

A society run makes roughly 240 model calls — six agents, dozens of ticks. In JSON mode the action vocabulary rides along as a ~450-token prose contract. In tool mode the equivalent tools schema is ~1,000 tokens, and — this is the load-bearing sentence — it is re-sent on every single one of those 240 calls. The schema is pure repeated input tokens. The most expensive seat in the house, the qwen3.5-plus commander, ate 59% of the run’s cost on its own; the five flash workers split the rest, their prompts dominated by that resent schema.

I genuinely tried to schema-trim my way out of it. I projected every option: strip the auto-generated pydantic title/default noise, compact the descriptions, even gut them to empty strings. The floor — descriptions deleted entirely — is about $0.069/run. Still above the JSON path’s $0.042, and still above the single-qwen3-max-does-everything baseline (~$0.06). You can’t trim a per-call tax down to nothing when you pay it 240 times.

The decision: default off, available and measured

So Aftershock ships with JSON contracts as the default — the cost-optimal path, and the one the published headline (“a society of small models matches a qwen3-max solo agent at ~35% lower cost”) actually reflects. Native function calling stays implemented, tested, and benchmarked, one flag away (aftershock run --arm society --society-tools), with its full ablation published in the repo and in docs/FIELD-NOTES.md.

I’ll defend that as the more sophisticated use of the API, not the less. Shipping the fancy feature by default because the rubric rewards fancy features is cargo-culting. Wiring it up, measuring it against a real baseline on identical worlds, discovering it doesn’t pay for itself in your regime, and making that an informed, reversible default — that’s engineering judgment. “We used native function calling and measured exactly what it costs” is a stronger sentence than “we used native function calling.”

The transferable lesson (with the scope caveat that matters)

The general rule: in a high-frequency multi-agent system, per-call overhead dominates — measure it before you adopt it. Anything you attach to every request (tool schemas, verbose system prompts, retrieved context, elaborate output formats) gets multiplied by your call count, and in a society that number is large. The fancier API is not free; it’s free per call and you make hundreds of calls.

And the caveat that keeps this honest: this is not “function calling is bad.” For a single-agent assistant, or any app that makes a handful of calls per task, a 1,000-token schema is a rounding error and the reliability and ergonomics of structured tool calls are well worth it — the math flips entirely. Function calling earns its keep at low call counts. It just gets taxed to death at high ones. Know which regime you’re in.

That’s the whole ethos of Aftershock in miniature: don’t ask whether the shiny thing can work — measure when it actually pays, on worlds identical enough that the answer means something. This time the shiny thing didn’t pay, and saying so out loud is the result.

Try it live: https://aftershock.redoubtlabs.dev · Read the code: https://github.com/bluntmachetti/aftershock

Built with Qwen Cloud (qwen3.5-flash / qwen3.5-plus / qwen3-max via DashScope) and Alibaba Cloud ECS, for the Qwen Cloud Global AI Hackathon.