Impact
Every applied fix is measured on the KPI it was meant to move, as a difference-in-differences against your own store trend — with honest verdicts, including "no change".
The question every CRO tool avoids is the only one that matters six weeks later: did that fix actually do anything?
Impact answers it. When a fix is applied, DynoWeb captures a baseline. After the lag appropriate to that kind of fix, it measures the same metric again and reports the before → after. Not a score, not a projection — a measured delta with its measurement window stated.
The right KPI per fix
Measuring every fix on revenue would be dishonest, because most fixes do not move revenue directly and the ones that do take a long time to show it. So each fix resolves to its own outcome metric through a registry — which metric, at which scope, which direction counts as better, and how long to wait.
| Kind of fix | Measured on | Better means |
|---|---|---|
| Frustration fix on an element | Rage / dead / error clicks on that selector | down |
| Hidden or weak CTA | Click-through rate on that element | up |
| Content below the fold | Scroll reach on that page | up |
| Form friction | Form completion rate, field abandon rate | up / down |
| SEO fix | Organic sessions to the fixed page | up |
| GEO / AI-search fix | AI referral sessions, or verified-live | up / binary |
| Speed fix | Field LCP p75 for that page | down |
| Conversion or pricing change | Conversion rate, revenue, channel revenue | up |
The scope is as specific as the fix. An element fix is measured on that selector, not on the page. A page fix is measured on that page, not store-wide. Measuring a button change against store-wide conversion rate is how tools manufacture both false wins and false failures.
Difference-in-differences
Why a plain before/after is not enough
Suppose you fix a CTA and its click-through rate rises 12%. If your whole store's engagement rose 11% that fortnight — a sale, a press mention, seasonality — your fix did almost nothing, and a naive before/after would have credited it with all twelve points.
Impact measures the change in the target and the change in the store-wide control over the same window, and reports the difference. What survives is the movement that is specific to what you changed.
This is not a randomised experiment and Impact does not claim it is. There is no synthetic control group and no counterfactual store. It is the strongest honest comparison available when the intervention has already shipped to everyone — which, for a theme fix or an SEO field, it necessarily has.
Windows and thresholds
| Setting | Value | Why |
|---|---|---|
| Baseline window | 28 days before the fix | Long enough for a stable baseline, short enough not to reach back into a different merchandising regime |
| Minimum sample | 50 | Below this, Impact reports "not enough data" rather than a number. A 3-sample lift is not a lift |
| Lag | Per KPI | A rage-click rate is readable in days; organic sessions are not. Each KPI carries its own settled-read delay |
| Provisional read | Some KPIs only | Where the metric comes from same-day telemetry, an interim read is taken early and clearly labelled provisional. Metrics with lagged external sources get one read, at the end |
A row is measuring until its lag elapses. Measuring rows are not blank — they
show the captured baseline, so you can see the "before" number DynoWeb is going to
compare against, on the day you shipped.
Verdicts
Impact reports what it found, including when what it found is nothing:
Improved
The target moved in the intended direction, beyond the store-wide trend, on enough sample to say so.
No change
The delta did not clear the bar. This is a real, common and useful result — it stops you scaling something that did nothing.
Regressed
It moved the wrong way. Shown as prominently as a win, because this is the one that saves you money.
Not enough data
Below the sample threshold. Honest rather than decorative.
Every figure is labelled an estimate. There are no guarantees anywhere in this surface, and that is deliberate.
Money, carefully
A $/mo revenue estimate appears only where there is a clean path from the
metric to money — conversion rate, revenue, channel revenue, or organic sessions
(traffic that can be valued at your own conversion rate and order value).
For a click-through-rate win or a rage-click reduction, no dollar figure is produced. It would require inventing a conversion assumption for the improvement, and that invented number would then be the headline. Instead those fixes report their real movement — "−42% rage clicks", "+18% CTR" — as secondary metric chips, alongside the primary verdict.
The monthly total at the top of Impact combines measured fix revenue with Popups incremental revenue — the holdout-normalised figure, not gross totals — so it reconciles exactly with the dashboard's earned-this-month headline and the Popups attribution view. Three surfaces, one number.
What enters Impact
- A suggestion you marked done — clickable straight back to the suggestion that proposed it
- A batch applied by SEO Autopilot
- Fixes verified by read-back, where "verified" means DynoWeb checked the value is live
Limits
These are properties of the method. Stating them is cheaper than arguing about them later.
- No counterfactual. You are compared against your own past plus a store-wide control, not a held-out group. External events inside the window — an algorithm update, a competitor's sale, a supply problem — are not controlled for.
- Seasonal stores are harder. A swimwear store fixing something in April and measuring in May is comparing across a demand ramp. The store-wide control absorbs some of that, not all of it.
- Concurrent changes share credit. Ship three fixes to the same page in the same week and the attribution between them is not separable. Space out changes you actually want to learn from.
- Small stores get "not enough data" often. The 50-sample floor is a real constraint. Saying nothing is the correct output when the data cannot support a claim.
- Proxies are labelled as proxies. Organic sessions stand in for search clicks; they are correlated, not identical. The registry records which KPIs are hard measures, which are proxies, and which are binary applied/not-applied.
- "No change" is sometimes measuring the truth about the apply, not the fix. If a change was marked done but never actually reached the live storefront, the metric will correctly report nothing happened.
Related
Revenue Attribution
Element-level revenue — the standing picture Impact's before/after is drawn from.
AI Suggestions
Where fixes originate, and where predicted impact is set before reality gets a vote.
Popups
Nudge revenue is incremental against a holdout, and joins the same monthly total.
Common questions
How long until a result is trustworthy?
It depends entirely on your traffic and the size of the effect. Impact will not declare a result until the affected traffic supports it, which on a low-volume store can mean weeks rather than days.
Is this the same as an A/B test?
Not quite. A/B testing splits traffic simultaneously; Impact compares comparable periods before and after. For nudges, DynoWeb does run true holdout splits — see Popups. Period comparison is the pragmatic option for theme and copy changes where a split is impractical.
What if I change several things at once?
Impact will tell you the combined effect and will not pretend to isolate individual contributions. If attribution per change matters to you, stagger them — the tool will say so when overlap makes a result unattributable.
Does it account for seasonality?
Comparison windows are chosen to be like-for-like where possible, and known distortions such as sale periods are flagged on the result rather than silently averaged in.
SEO Autopilot
Scans your whole Online Store for empty or weak SEO fields, writes them, and applies through the Admin API — never theme files, always reversible, verified by read-back.
Revenue Attribution
Which buttons, links and images on your store actually carry orders — per element, per page, per device, written from Shopify's own order webhooks.