We Audited Our Own Decision Engine
Most of what we write about is what happens when a marketing team trusts a number it shouldn't. This one is about us. We went looking for the same failure mode inside our own decisioning pipeline — the system that reads ad account data and tells a team whether to scale, hold, or kill a piece of spend — and found four ways it could quietly look right while being wrong.
None of these were catastrophic on their own. That's what made them worth writing up. Each one is the kind of thing that survives in production for a while precisely because it doesn't throw an error — it just produces a slightly wrong answer, confidently, in a way that's very easy not to notice.
Bug 1: A confidence score that could never actually be low
The system has a rule that's supposed to protect against overconfident recommendations: if the model's confidence in a call falls below a threshold, don't ship that call as a firm verdict — flag it as uncertain instead. Reasonable design. The problem was in the plumbing: the confidence value feeding that check was hardcoded to a constant, the same number, every single time, regardless of how much evidence actually supported the call underneath it.
The gate itself worked exactly as designed. It just never had a real number to work with. Every recommendation looked equally confident — the well-supported ones and the shaky ones alike — because the one thing that was supposed to distinguish them from each other had been quietly disconnected from reality. A safety check that can't fail isn't a safety check. It's decoration.
Bug 2: A viability floor tested on the wrong basis
The system has a kill floor — a minimum return threshold below which an account gets flagged as genuinely underwater and a stop-spending recommendation gets triggered. The floor test itself is fine. The issue is what it was tested against: a version of the account's return that included revenue attributed from channels outside the ads being judged — marketplace halo effects, cross-channel lift, that kind of thing.
That's real revenue, and it's fine to report. It is not the same thing as asking "is this specific spend, on these specific ads, earning its keep." An account that was genuinely losing money on its ads could pass the kill check anyway, propped up by credit that had nothing to do with the ads themselves. The two questions — "is the business doing fine overall" and "is this spend specifically worth continuing" — got merged at exactly the point where they most needed to stay separate.
Bug 3: A peer benchmark built from a failing cohort
Some recommendations get checked against a peer group — is this account behaving like the other, similar accounts around it? That's a legitimate sanity check, but only if the peer group itself is healthy. We found a case where the majority of accounts in a comparison cohort were themselves below their own individual viability floors.
"In line with the peer median" is only reassuring if the peers are doing well. When most of the comparison group is quietly failing, being in line with the median just means being in line with everyone else's problem. The check passed. It just wasn't checking anything.
Bug 4: A narrative that read a stale window
The system also generates a written summary alongside each recommendation — a plain-language explanation of what it saw. One of these called an account "stable" during a period where its performance had actually been declining for four straight days. The summary wasn't wrong about the data it was reading; it was reading the wrong window. It was built off a rolling multi-week average, long enough that a real, recent break in performance gets smoothed into invisibility before the narrative ever sees it.
The underlying daily numbers already showed the decline clearly. The summary just never looked at them at the right resolution.
What ties these together
None of these four bugs are exotic. Each one is a version of the same basic mistake: a check that's supposed to answer a hard question quietly ends up answering an easier, more flattering one instead. Is this call confident? (Well, the number says so, because the number never changes.) Is this account viable? (Well, it clears the bar, if you let outside credit count.) Is this account normal? (Well, it matches its peers, if the peers are also struggling.) Is this account stable? (Well, the long-run average says so, if you don't look at the last four days.)
That pattern is worth naming because it's not specific to marketing decisioning. Any system that grades outcomes — a monitoring dashboard, a risk model, a QA process — can develop the same quiet drift, where a check keeps running and keeps returning a green light, but the thing it's actually testing has slipped away from the thing you meant to test.
What we changed
The fix, in each case, was the same shape: make every gate answer the harder question first, not the more convenient one that happened to be lying around.
Confidence has to be computed from real inputs before it's allowed to suppress anything. Viability gets tested against the return the spend itself generated, before outside credit is allowed to count. A peer comparison only gets to speak once the peer group has been checked for its own health. And any narrative-generating step reads the recent window at full resolution before it's allowed to reach for a long-run average.
We're writing this up publicly for the same reason we built the checks in the first place: a decision system is only worth trusting if it's willing to be checked itself, on the same terms it applies to everything else. The system in question was already "working," in the sense that it ran, produced outputs, and mostly looked reasonable. It just wasn't asking the right question first, four different ways.
Worth checking whether yours is too.