Econometricians have a specific term for a moment where the underlying relationship inside a dataset genuinely changes, and data from before that moment can no longer be safely compared to data after it as though nothing happened: a structural break. It's usually discussed in the context of financial markets or major policy shifts, a central bank rate change, a recession, a regulatory reform. But the same statistical problem shows up constantly in marketing data, most often triggered by a CRO test, and almost nobody names it when it happens.
A page redesign, a form restructure, or a change to what counts as a completed conversion doesn't just change the page. If it changes what a metric is actually measuring, it creates a structural break in that metric's history, even though the dashboard keeps drawing one continuous line straight through it.
What a structural break actually is
The core idea is simple: a time series is only meaningfully comparable across its full length if whatever it's measuring stayed consistent throughout. When something changes the underlying process, a new law, a market shock, a redefinition, the relationship between the data before and after that point can genuinely differ, and treating the whole series as one continuous, comparable line produces conclusions that don't actually hold.
Econometricians test for this explicitly before drawing conclusions from long time series. Marketing teams almost never do, because the tools don't flag it automatically, and because a conversion rate chart looks exactly the same whether the metric's definition held steady or quietly shifted halfway through.
A structural break isn't a data quality problem, the numbers on both sides can be perfectly accurate. It's a comparability problem: the two halves of the series are answering slightly different questions while wearing the same label.
Why the metric name staying the same hides the problem
This is what makes structural breaks in marketing data so easy to miss. "Conversion rate" is still called conversion rate before and after a form redesign. Nobody renames the metric, because from a dashboard's perspective, nothing changed, the same event still fires, the same field still populates the same report.
But if the redesign changed what triggers that event, added a qualifying step, removed a field that previously filtered out a segment of visitors, or altered where in the flow the "conversion" moment is counted, the metric is no longer measuring quite the same thing it was before. A rising or falling line on a chart that spans the change looks like a genuine trend. It may just be two different measurements stitched together as though they were one.
Where CRO work most often creates a break
Three specific moments in a typical CRO programme create breaks more often than teams expect, precisely because each one feels like a small, contained change at the time.
An A/B test declares a winner and it becomes the new default. If the winning variant changed what counts as the conversion event, not just how it looks, the historical baseline it was tested against is no longer directly comparable to performance afterward.
Redesigns often coincide with analytics or tag manager updates. A tracking change bundled into the same release as a page change makes it genuinely difficult to tell later whether a shift in the data reflects the design or the measurement.
A "lead" gets redefined informally over time, a marketing ops change here, a sales feedback loop there, without a single formal changelog entry marking when the definition shifted. Months later, nobody can say with confidence which side of the chart is measuring which definition.
Documenting and bridging a break instead of ignoring it
None of this means CRO tests should stop changing conversion events, that would defeat the purpose of testing. It means the change needs to be treated as a genuine break in the record, not quietly absorbed into a continuous chart.
- Log every change to what a conversion event measures, not just what the test changed visually: A simple changelog, date, what changed, what the event now captures, turns a silent break into a documented one that future analysis can account for.
- Run old and new tracking in parallel where possible: A short overlap period where both the old and new conversion definitions fire lets you establish the actual relationship between them, rather than guessing at it after the fact.
- Annotate charts at the break point, don't just let the line run through it: A visible marker on the dashboard prevents a stakeholder from reading a definitional shift as a genuine performance trend.
- Treat pre- and post-break periods as separate series when reporting long-term trends: Compare rates of change within each period rather than drawing a single trend line across a point where the underlying measurement changed.
- A structural break is a point where what a metric measures genuinely changes, making before-and-after data unsafe to compare directly
- CRO tests create these breaks constantly, most often through field changes, new qualifying steps, or redesigned multi-step flows
- The metric's name staying the same on the dashboard is exactly what makes the break easy to miss
- Log every conversion event change, run old and new definitions in parallel where possible, and annotate charts at the break point
Our UX audits map real navigation paths against the wayfinding principles here, decision points, confirmation, and landmarks, so fixes target where visitors genuinely lose their way rather than a guess at it.
Explore UX Audits →