Every CMO eventually gets to this question. 

Sometimes it’s the CFO who asks it first, in a budget review, with a straight face:
“How do we know this AI is actually driving revenue, and not just taking credit for sales that would have happened anyway?”

It’s a fair question. And it’s harder to answer than most vendor decks make it look.

BrazeAI Decisioning Studio is designed to make individualized marketing decisions while measuring performance against comparison groups. Because the system continuously evaluates different actions, performance needs to be assessed with the appropriate control, KPI, and reporting context rather than by looking at engagement metrics alone.

The measurement framework is part of the product experience. Braze describes Decisioning Studio as measuring uplift against random control, holdout, or business-as-usual groups, while its reporting surfaces performance and variant insights. The exact controls and metrics available depend on the Decisioning Studio edition and configuration.

Why opens, clicks, and attribution numbers lie to you 

Opens and clicks are engagement metrics, not necessarily causal business outcomes.

A message can get opened, clicked, and still do nothing for revenue. Someone was going to buy the product anyway. Someone clicked out of curiosity and bounced. The engagement happened, but the business outcome you actually care about didn’t move because of it.

Attribution and incrementality answer different questions. Attribution assigns credit among touchpoints according to a chosen attribution method; incrementality asks what additional outcome occurred because of an intervention compared with an appropriate counterfactual or comparison group.

So the stronger measurement question is: “What changed because of the AI decision compared with an appropriate comparison experience?” That is the purpose of incrementality measurement.

How Decisioning Studio’s control groups actually work

Decisioning Studio includes comparison mechanisms, although the exact mechanism depends on the product edition and setup.

For Decisioning Studio Go, Braze documents a required random control group alongside the AI-optimized group. The random control receives randomly selected combinations of the same configured options on randomly selected days. For broader Decisioning Studio programs, Braze also describes measuring uplift against random control, holdout, or business-as-usual groups.

In Decisioning Studio Go, the random control is ongoing and has a minimum size of 5% of the selected segment. Braze says this control receives randomized combinations of the configured options, allowing performance to be compared with the AI-optimized group continuously.

It’s important not to conflate different control designs. A Decisioning Studio random control is tied to the decisioning experience and compares AI-optimized choices with randomized choices. A broader messaging holdout or global control answers a different measurement question about the incremental impact of messaging overall. Use the control that matches the question you are trying to answer.

Take a win-back campaign as an example. If the agent is evaluated against an appropriate control or business-as-usual group, the difference in the defined outcome provides evidence of incremental performance attributable to the decisioning experience, subject to the quality of the experiment and measurement design.

Incrementality vs A/B testing vs Experience optimization

Decisioning Studio is related to experimentation, but it is not simply a conventional A/B test.

A conventional A/B test generally compares predefined variants for a defined experiment. Braze describes Decisioning Studio as running continuous experimentation and learning, with agents adapting their choices as new engagement and outcome data becomes available. Decisioning Studio Go, for example, continuously learns from ongoing engagement feedback.

Incrementality is the measurement concept underneath a causal comparison: it estimates the additional outcome associated with an intervention relative to an appropriate control or counterfactual. Raw conversion or revenue totals do not, by themselves, establish that causal lift.

Incrementality is an established marketing measurement discipline that predates AI decisioning. Decisioning Studio applies comparison-based measurement to AI-optimized customer decisions and experiences.

Take a cross-sell agent as an example. If its Incremental LTV per Customer number is climbing, that’s a start. But the story only gets useful once you know why it’s climbing.

The metrics that actually matter in the performance report

This is where most people get lost, so let’s break it down.

Performance reporting should be read in the context of the selected date range and KPI definition. For example, a full-period KPI should not automatically be interpreted as the simple average of daily KPI values.

When reading an uplift figure, confirm the dashboard’s definition of the primary and comparison groups and the KPI being used. Do not assume a formula or implementation detail unless it is documented for the specific Decisioning Studio report or release.

Hierarchical or driver-style breakdowns can help explain which underlying components or dimensions are associated with movement in a top-line KPI. Use the definitions shown in the specific report to understand exactly how each component is calculated.

Keep volume and rate metrics separate. Total conversions or revenue describe the amount of activity or value; conversion rate and revenue per user describe efficiency or value per customer. They answer different questions and should not be interpreted as interchangeable.

Back to that cross-sell agent. Say its Incremental LTV per Customer card is trending up, and the driver tree shows the lift is coming from conversions per customer, not revenue per conversion. That tells you the agent is reaching more converters, not just squeezing more out of the ones it already had. That’s a meaningfully different win, and only the driver tree tells you which one you got.

Reading the report without fooling yourself

A number on a screen isn’t proof. Reading it correctly is.

Start with sample size and statistical uncertainty. Small or early samples can produce unstable estimates, so define the decision rule and significance threshold before interpreting trends where statistical inference is being used. Do not treat a visually positive trend as proof on its own.

Also distinguish daily trend values from full-period metrics. A full-period conversion rate is normally calculated from the underlying aggregate counts, not by taking a simple average of daily conversion rates. As a result, a period-level metric can differ from the arithmetic average of daily percentages.

Check data completeness before interpreting very recent performance. Processing or activation delays can make the latest period incomplete, so confirm that the reporting window has finished processing before drawing conclusions.

Explainability: Why the agent chose what it chose

Knowing that something worked is only half the story. You also need to know why.

Agent Preferences reporting shows how frequently the agent selects available variants or options. This can help teams understand what the agent is favoring and whether the distribution of choices changes over time.

If SHAP-based explainability is available in the relevant Decisioning Studio implementation, SHAP values can be used to analyze how model features contributed to a prediction or recommendation. Treat this as model explanation, not as a complete causal explanation of why a business outcome occurred.

Put performance measurement and model insights together, and you get a more complete view: comparison-based measurement helps assess whether the decisioning experience is producing incremental value, while preference and explainability views can help teams understand what the agent is doing and which inputs are associated with its decisions.

Wrapping up

Proving AI decisioning is working requires three complementary layers: an appropriate comparison design, an outcome-focused measurement approach that distinguishes incrementality from attribution, and reporting that helps explain both the size of the result and the factors associated with it.

AI decisioning should be evaluated with disciplined controls, clearly defined KPIs, complete reporting windows, and appropriate statistical interpretation, not engagement metrics alone.