Your A/B test didn't fail. It was never a test.

• Your test didn't fail. It was never set up to tell you anything. That is why so many test programmes run for years and teach the team nothing.
• Below: the six mistakes that ruin experiments, one sentence that fixes most of them, a worked example and a simple test log.

WHAT WE MEAN BY A TEST

CRO (conversion rate optimisation) is the work of getting more of your visitors to buy, sign up or book. The main tool is the A/B test: show version A to some visitors and version B to others, then compare.

In SEO, tests often work differently: you change a group of pages and compare them with the period before, or with similar pages you left alone.

Giulia Panozzo, a neuroscientist turned marketer and founder of Neuroscientive, wrote about the six mistakes that ruin SEO experiments in Search Engine Journal this week. We read it as CRO people, and every mistake applies to landing pages, checkouts and ads too.

MISTAKE 1: NO HYPOTHESIS THAT CAN BE WRONG

"Improving page titles will improve SEO performance" sounds like a hypothesis. It isn't. "Improving" and "performance" are not defined, so no result can prove it wrong.

A real hypothesis predicts something the data could contradict. Panozzo's example: add a country reference to the titles of 700 pages and predict 10% more clicks over four weeks, compared with the four weeks before. A 12% lift supports it. A 9% lift falsifies it. Ask yourself one question before launch: what result would prove us wrong?

MISTAKE 2: NO CLEAN COMPARISON

Your test is not the only thing happening. A site-wide feature launch, a sale, a change in demand, a Google update or a developer editing the same pages can all move your numbers. These are confounding variables: other causes that look like your effect.

Panozzo admits she scrapped one of her own study designs for exactly this reason. The fix is a change log: write down every internal release, every promo, every holiday and every outside event, with dates, so you can see what else moved during the test.

MISTAKE 3: DECIDING WHAT SUCCESS MEANS AFTER THE RESULTS

If you choose the metric after you see the data, you will find a winner every time. "Sales didn't move, but impressions did." That is confirmation bias with a dashboard.

Write it down before launch: the main metric, the time period, what you compare against and what counts as evidence. Then a test gets judged, not defended.

MISTAKE 4: DRAWING CONCLUSIONS THE TEST CAN'T SUPPORT

A test has three possible outcomes, not two: evidence of an effect, evidence against the effect you predicted, or not enough evidence. Short tests, small samples, noisy data and small effects often land in the third box.

As Panozzo puts it, absence of evidence isn't necessarily evidence of absence. And timing matters: she notes that a seven-day window doesn't necessarily mean seven meaningful days of exposure. More data won't rescue a badly designed test.

MISTAKE 5: TREATING EVERY TEST AS A ONE-OFF

One test is one sample, one method, one period. Confidence comes from repeating results and stacking evidence. Panozzo recommends logging the question, hypothesis, change, sample, dates, context, result, interpretation, limits and next step for every test, including losers and inconclusive ones. In her words, about 20 properly documented tests start telling you something about your site.

MISTAKE 6: TESTING FOR TESTING'S SAKE

Before a test, finish this sentence: "Depending on what we find, we will..." If nothing would change, don't run it. Her example: a change that improves organic clicks by 0.3% but takes six months of engineering work. Real, and still not worth it.

THE SENTENCE THAT FIXES MOST OF IT

"We change X. We expect Y to move by Z% within N weeks. If it moves less than W%, we drop it. Depending on the result, we will ___."

A worked example. Say your checkout converts 3% of visitors. You write: "Showing delivery dates on product pages lifts completed orders by 8% within four weeks. If the lift is under 3%, we drop it. If it wins, we roll it out to every product page." Now the test can lose, the metric is locked, and the decision is already made.

THE PUNKD TAKE

Most teams don't have a testing problem. They have an honesty problem. They run tests to defend ideas they already love, then dig through the dashboard until something turns green. A test that can't lose isn't a test. It's a press release.

The fix costs nothing: five minutes of writing before launch. Teams that do it learn something from every test, including the ones that lose.

STEAL THIS

1. Write the prediction, the main metric and the kill line before every launch.
2. Keep a change log of releases, promos, holidays and algorithm updates.
3. Label every result win, loss or inconclusive, and never report inconclusive as a loss.
4. Store every test, losers included, in one shared log with the same fields.
5. Finish "depending on the result, we will..." before you spend a day building anything.

When did your team last write a kill line before a launch?