Stop Asking Your AI Whether to Cut the Campaign

"Should we cut this campaign?" is the question most marketing teams ask an AI tool first. It is also the worst place to start.

It bundles five jobs into one answer: read the context, check the facts, explain the performance, estimate what cutting would do, and then choose an action. A model that answers with a confident "pause" has skipped most of that work without telling you.

Decision models like Jev are interesting for a narrower reason. They return typed judgments and probabilities that software can use directly, instead of a free-form recommendation. Used well, they can review far more of your account than your team has time to read. Used badly, they turn a guess into a budget change.

This guide is a practical way to tell the difference. It covers the five jobs, how to start with one judgment, where to put the gate, what "deterministic" has to mean, what fine-tuning can and cannot teach, and how to measure the whole workflow before you trust it with money.

What a decision model actually returns

A Jev-like decision model takes a piece of context and returns a structured answer: which category fits, with a probability, from options you define. In marketing, the context is usually an agency note, a merchant instruction, or a weekly campaign review.

The output is a judgment about the context. It is not a fact about the business, and it is not a forecast of revenue. Keeping those three apart is the most useful habit in this guide.

The five jobs in marketing decisions

Decision work breaks into five roles. Vendor pitches often blur them together, so keep them separate.

1. Semantic (Jev-like). Which answer fits the supplied information? Example: does this agency note report a measurement change? It does not establish that the change happened, or how big it was.
2. Predictive. What outcome is likely under stated conditions? Example: expected conversions next week if spend holds. It does not establish what changes if we intervene.
3. Causal. What changes because we intervene? Example: incremental sales from a spend cut. It does not establish which action is best.
4. Optimization. Which feasible action best serves our objective? Example: a budget split across channels under a margin floor. It does not establish whether the objective is right.
5. Policy learning. Which action rule performs well across contexts? Example: when to pause a creative after a promotion ends. It does not establish that the rule holds for a new brand.

These roles overlap in real products. They are not five products you need to buy. The point is to know which job a tool is doing when it claims to "make decisions."

Start with one judgment, not a budget decision

The fastest way to get value, and the fastest way to get hurt, is to hand a model the budget decision directly. Start instead with a judgment that a person currently has to read.

Take this review note from a hypothetical campaign:

The promotion ended Sunday. Two variants are unavailable. Purchase events became intermittent after the checkout update.

A useful first question is not "should we cut spend?" It is:

Does this note report a change to measurement?

Define the answers before you test anything:

• Yes, measurement changed
• No, measurement is stable
• Unclear

Keep the "unclear" option. A model forced to choose between yes and no will be confidently wrong on the notes that matter most.

Then test it on real reviews. Pull 100 to 200 past notes, have two people label them independently, and check where they disagree. The labels that agree become your test set. The disagreements show where your definitions are weak.

Put the gate between the judgment and the money

This is where most teams go wrong. A well-formed judgment does not authorize an action.

In the example above, the model can read the note correctly, and an allocator elsewhere in the stack can still propose more spend on the variants that are running. Each output looks reasonable on its own. Together they can do real damage.

The fix is a policy that sits between them. For example:

1. If a note reports an unresolved measurement change, spend increases on the affected campaigns are held.
2. The case goes to an investigation queue, not to the budget API.
3. A person resolves the measurement question before the hold is lifted.

The rule is simple. The model can change what your team looks at. Only an explicit policy can change where money moves.

Every case should leave an audit record with four things: the original evidence, the model's output, the policy that blocked or allowed the move, and what actually happened next. A fluent explanation written after the fact is not a substitute.

What "deterministic" has to mean

Teams often ask for a "deterministic" model. That word hides three different properties, and they need separate answers.

1. Structural validity. The output matches the declared type and allowed values. This is cheap to check, and it does not make the answer true.
2. Repeatability and replay. Given the same inputs, definitions, model version, and serving conditions, you can reproduce the decision. Store the model version, the prompts, the calibration settings, and the actual outputs. Replaying a recorded judgment through a fixed policy does not require a hosted model to return identical probabilities later.
3. Enforced authority. The execution layer checks current limits and permissions, rejects stale proposals, and blocks duplicate execution. A model's confidence cannot override those checks.

The first property is cheap. The third protects your budget, and it is code you write. It is not a property of the model.

Formal verification research makes the same split. Anthropic's recent work on formally verifying software says it plainly: a proof checker can confirm that a program meets its specification, but it cannot tell you whether the specification matches reality. A marketing judgment has the same gap. Passing a structural check proves the answer is well formed. Whether the answer is right depends on the definitions behind it.

Fine-tuning teaches conventions, not profit

If your team's past approvals are the training data, a fine-tuned model may get very good at agreeing with your team. That is different from learning which decisions make money.

Look at what each kind of label teaches:

• "This note reports a tracking concern" teaches interpretation.
• "The marketer approved a budget cut" teaches a preference or convention.
• "Profit rose after the cut" records an outcome. It does not isolate the cut's effect.

Only the third label touches profit, and it is the weakest one. Profit moves for many reasons, including seasonality, inventory, and other campaigns. Learning the effect of a cut needs a credible comparison with what would have happened otherwise.

Before you fine-tune, make the task definitions and the current business context explicit. If the model still misreads a recurring judgment, that is a specific target for adaptation. Then evaluate on held-out reviews, including new time periods and, where relevant, new brands. Fit any recalibration on separate data, and check calibration on untouched test reviews. Better labels do not guarantee better confidence estimates.

Measure the whole workflow, in shadow mode first

Model latency is the wrong thing to measure. Measure the workflow:

• Total review cost per account per week
• Time from a flagged note to a correct disposition
• Human review load, including false alarms
• Missed reports of real measurement changes
• The cost of the errors that get through

Run the system in shadow mode first. It produces judgments and proposed actions, but spend does not change. Compare its dispositions with what your team actually did.

Any claim that the system improves profit then needs a controlled live comparison, an agreed outcome window, and hard exposure limits. Two clocks run here. A model can interpret new context in seconds. Conversions and returns take days or weeks to mature. Faster classification is not a reason to keep reallocating against immature evidence.

When a rule beats a model

If the fact is already a clean structured field, use a rule. A stock count below a threshold does not need a language model.

Reserve the model for context that lives in free text: agency notes, merchant instructions, campaign reviews, and the commentary that never makes it into a dashboard. The gain is consistent, bounded judgments across that text, so more of your account becomes reviewable without someone reading every note.

A practical checklist

Before a decision model touches a live account, check that:

• Each judgment has a written question, a closed list of answers, and an "unclear" option.
• Labels come from at least two people who labeled independently, with disagreements resolved.
• Every model output is stored with its version, inputs, and timestamp.
• A policy, not the model, decides whether spend can move.
• Known measurement changes put affected campaigns on hold until a person resolves them.
• Exposure limits exist per campaign and per day.
• Shadow mode ran long enough to surface the error types that matter.
• A held-out test set includes a new time period.
• You measure review cost, missed flags, and false alarms, not only accuracy.
• Someone owns each policy rule and reviews it on a fixed schedule.

How we think about it at Niti

We run decision work through five steps: detect, explain, recommend, gate, measure. Judgment models help with the first two, by spotting a leak and explaining the evidence behind it. The gate and the measurement loop are where the risk sits, because that is where a wrong call gets stopped and a right call gets checked against outcomes.

The question to ask any vendor is not "does it make the call?" It is "which of the five jobs is it doing, and where does the gate sit?"