How to Benchmark Against Yourself Over Time
A like-for-like baseline is defined as a reference level built from a store's own prior performance on a fixed segment definition and a fixed comparison period, so a later measurement differs from it only because of changes the store made. That is what you need in order to benchmark against yourself over time, and it is why a category average cannot do the job. Around 70% of carts are abandoned across eCommerce in any normal month [Baymard Institute, 2025], so the interesting question is never the level but the direction on comparable traffic. This guide covers how to build the baseline, which four measures to hold it on, and the comparison errors to avoid.
- Write your segment definitions down, because an undocumented definition drifts quietly.
- Choose one comparison period and keep it, including in a quarter you dislike.
- Hold four measures together: conversion rate, revenue per visitor, order value and repeat rate.
- Compare each segment against itself, never one blended average against another.
- Keep a dated change log beside the numbers, or a difference has no cause.
- Read monthly and judge quarterly, since weekly readings mostly show variation.
Omniconvert has measured store performance across the CROBenchmark dataset of 7,000+ websites in 15+ industries, against 248+ audit criteria, over 13 years in eCommerce [CROBenchmark Report 2026, Omniconvert]. Stores with a documented internal baseline are a minority of that set, and they are the ones able to say whether last quarter's work paid for itself.
Why you benchmark against yourself, not a category average
This is not an argument against external benchmarks, which are genuinely useful for setting expectations and for sanity-checking a number that looks wrong. It is an argument about which question they answer. "Is 1.9% a normal conversion rate for my category" is a benchmark question. "Did the work we did in the last quarter make the store better" is not, and no external average can answer it.
The reason is that the average is a composite of businesses whose traffic you cannot see. Two stores with identical conversion rates can differ wildly in where their sessions come from, which is the argument made at length in benchmarks versus your own baseline.
So the useful comparison is you, earlier, on comparable traffic. That sounds obvious and it is surprisingly hard to do, because the comparison has three moving parts and most teams fix none of them.
The three things you have to fix first
Segment definitions come first. Write down exactly what counts as new and returning, which channels sit in paid and which in organic, whether app traffic is included, and what is excluded entirely. These seem too obvious to write down, which is why they drift: an attribution setting changes, a channel gets regrouped, and a year later you are comparing two different populations.
The comparison period comes second and is the one most often corrupted. Pick the same phase of the previous cycle, write it down, and keep it when the number is unflattering. A baseline that moves to suit the story has stopped being a baseline, and that failure is almost never deliberate.
The change log comes third and costs the least. A dated list of releases, campaigns, price changes and promotions, kept beside the numbers. Without it a baseline can show you a difference and never tell you its cause, which is the state most reporting is in.
The quiet version of this failure: a team sets up a careful quarterly baseline, then six months later the analytics channel grouping is adjusted so that one email tool's traffic moves from referral into direct. Every subsequent comparison shows organic strengthening and referral collapsing, a story gets built around it, and nothing about the store changed at all.
The four measures to hold together
| Measure | What it isolates | How it misleads alone |
|---|---|---|
| Conversion rate | Share of sessions that bought | Falls whenever cheap traffic is added |
| Revenue per visitor | Value of a session | Hides a basket change as a rate change |
| Average order value | Basket composition | Rises on discount withdrawal, not improvement |
| Repeat purchase rate | Whether growth is earned | Lags a change by a full purchase cycle |
Revenue per visitor is the single most useful of the four if you are forced to pick one, because it absorbs both the rate and the basket. It is still not sufficient: a rise driven entirely by price increases reads identically to one driven by better merchandising, and only the order value column tells you which you are looking at.
Repeat purchase rate earns its place because it is the measure that distinguishes buying growth from earning it. It also moves slowly, so treat it as a quarterly measure and do not expect it to confirm anything within a month. The general case for weighting retention this way is well established: a 5% retention lift can raise profits 25% to 95% [Harvard Business Review, 2014].
AliveCor used Omniconvert to run a structured A/B testing programme and achieved +21% conversion rate, +5% revenue per visitor, and 94% statistical relevance across their experiments. [Omniconvert, AliveCor case study] Notice that two of the four measures moved by different amounts, which is exactly why a single-metric baseline cannot describe a result.
Normalising for traffic mix
The mechanics are simple once the segment definitions are fixed. Paid against paid, organic against organic, mobile against mobile, new against new, each against the same phase of the previous cycle. Then look at whether the segments moved in the same direction, because that is the thing an average destroys.
Two patterns are worth naming. When every segment improves and the blended rate falls, you added traffic: that is a growth story being misreported as a decline. When the blended rate holds and one segment collapses, you have a real problem hidden by a favourable mix shift, which is the more dangerous of the two because nothing in the headline number asks for attention.
Where you do want external context, Shopify's own writing on conversion rates is clear that a reported figure depends on how sessions are counted, which is the same reason internal definitions have to be fixed before anything is compared.
What to do this week
- Write the baseline definition. One page: segment definitions, the comparison period, the four measures, the review cadence, and who owns it. Circulate it before the next review.
- Start the change log today. Dated releases, campaigns, price changes, promotions. Even a shared spreadsheet beats reconstructing a quarter from memory.
- Re-cut your last two quarters by segment. Against the fixed period. Expect at least one story you have been telling to turn out to be a mix effect.
Where the segment comparison shows a genuine decline, a free CROBenchmark audit is the quickest way to see which of the 248+ criteria the store is failing on the affected pages, and it is the right step before committing testing budget. Omniconvert Explore is the platform for the testing that follows, with A/B and multivariate testing, on-site surveys and session recordings; Explore averages a 23.2% conversion uplift across 70,000+ experiments. Further upstream, Omniconvert Commander (formerly Nexus) is an AI for eCommerce growth engine that unifies commerce data, prioritises experiments by True Profit, and generates campaigns and creative you approve before they go live.
One note on how this reads to a stakeholder. A like-for-like baseline will often make your numbers look less impressive than a blended one, because it strips out the mix effects that were flattering you. That is the point, and it is worth saying out loud the first time you present it, before somebody else notices and asks why the figures changed. The sequencing from a confirmed gap to a test plan is covered in how benchmarks should shape your test roadmap, and the diagnostic order underneath it in the audit it checklist.
The companion piece on this is prioritise fixes when you're below benchmark.
Frequently asked questions
How do you benchmark against yourself over time?
Fix your segment definitions and your comparison period, then measure each segment against its own prior level rather than against a blended average. Hold four measures together: conversion rate, revenue per visitor, average order value and repeat purchase rate. Keep a dated log of what you changed beside the numbers, and judge the result quarterly rather than weekly.
Why not just use an industry benchmark?
Because a category average hides the traffic mix, the price point and the product range that produced it, none of which match yours. It is useful for judging whether a number is plausible and useless for judging whether you improved. Your own prior performance on a fixed segment definition is the only comparison that isolates the effect of what you did.
Which measures should the baseline hold?
Four, together. Conversion rate tells you how many sessions bought, revenue per visitor tells you whether the sessions got more valuable, average order value separates basket change from conversion change, and repeat purchase rate shows whether you are buying growth or earning it. Any single one of the four can mislead, and the set of four rarely does.
How do you handle a changing traffic mix?
By never comparing blended averages. Compare paid against paid, organic against organic, mobile against mobile, new against new. A blended rate falls whenever you add cheaper traffic and rises whenever you cut it, so a blended comparison mostly measures media buying decisions rather than store performance.
How often should you read it?
Monthly to read, quarterly to judge. A weekly reading on a normal store is dominated by variation rather than by change, and teams that review weekly end up reacting to noise and rewriting pages that were working. Set the cadence before the first review, because a cadence chosen during a bad week will always be too short.
What is the most common mistake?
Moving the comparison period when the number is unflattering. It usually happens without anyone intending it: a report switches from year-on-year to the trailing month because the trailing month looks better, and after two or three such switches nobody can say whether the store improved. Write the comparison down and keep it even in a bad quarter.
Write the one page first
The baseline is a document before it is a dashboard. Segment definitions, one comparison period, four measures, a review cadence and a named owner, on a single page that everybody who reads the numbers has seen.
Then start the change log. A difference without a dated cause beside it is a conversation rather than a finding, and that one habit is what turns a reporting ritual into evidence.