Headline A/B test: how one headline lifted conversion rate 116%

Why test the headline first?

The headline is the one line of copy almost every visitor reads. Most of the page is skimmed: on an average web page, users have time to read at most 28% of the words, and 20% is more likely (Nielsen Norman Group). On a paid landing page the headline also has a second job: it confirms the promise the ad made, in the first second, before the visitor decides to scroll or leave.

Small headline changes can carry big money. At Bing, a headline change that sat in the backlog for months was finally A/B tested, lifted revenue by 12% and became the best revenue idea Bing ever had, worth $100 million (Harvard Business Review).

That is why headline testing sits near the top of our test queue for any page with steady paid traffic. It is cheap to build, it touches every visitor, and it does not need a developer or a new design.

How we set up the headline A/B test on Shopify

The page is the brand's main paid landing page on Shopify, a bundle offer page that takes cold and warm traffic from paid social. We changed one element: the H1 text.

1. A text edit, not a new page. Each arm swaps the H1 text in place in Intelligems. The URL and template stay the same, so nothing but the words can explain a difference.
2. Paid visitors only. The audience is visitors who arrive from a tagged paid ad. Organic visitors, internal clicks and crawlers without ad parameters are left out, so bots do not dilute the read.
3. One page. The test only runs on this landing page, and no other test on the page touches the same visitors.
4. Five arms, 20% each. A control and four challengers. Five arms need more traffic than two, so we only do this on a page with steady volume.
5. A scoreboard agreed before launch. Revenue per visitor and conversion rate decide it. Average order value and abandoned checkout rate are guardrails: a headline that lifts conversion but fills checkout with buyers who leave is not a winner.

The five headlines: which angle won?

We wrote each challenger on a different angle, not as word swaps of the same line. The real headlines stay private, so the structures below use brackets for the product-specific words.

Control: deadline promise

The control was the winner of round one. It makes a strong promise with a deadline and removes the biggest objection, that the buyer has to give something up. It beat the brand's original headline by 41%. Its weakness: it leads with the problem word and makes an absolute claim, and skeptical cold visitors discount absolute claims.

H1: pain removal (+24%)

Names the exact moment the buyer lives with the problem every day. Specific and relatable, so it beat the control. But it still sells the absence of something, and the absence of a problem is hard to picture.

H2: felt outcome (+116%, the leader)

A time window, the word relief, and the feeling the buyer wakes up to. It never repeats the problem; it sells the after-state the buyer wants to feel. It is also the most believable line in the test: relief and a more comfortable moment are promises a skeptical visitor can accept, where "gone" invites an argument.

H3: benefit stack (+25%)

Two benefits in one line. Clear and better than the control, but two promises split attention and each one lands smaller than a single, concrete outcome.

H4: ritual reframe (-9%)

The clever one. It reframes a daily routine as something to enjoy. It was the only headline below the control and had the worst abandoned checkout rate in the test. Clever copy needs a second read to land, and paid social visitors give a headline one read.

What did the headline test results show?

The read below is about 19 hours into round two, with about 2,100 US visitors split across the five arms, roughly 400 to 450 per arm. Here is each metric, one panel at a time.

Conversion rate: +115.88%

The felt-outcome headline converted 3.73% of visitors against 1.73% for the control, a 115.88% lift. Three of the four challengers beat the control. Only the clever reframe fell below it. Same traffic, same page, same price: the only thing that changed was one line of text.

Revenue per visitor: +118.98%

Revenue per visitor is the number we decide on, because it combines conversion rate and order value. H2 earns 2.19 times the control per visitor. The ranking matches conversion rate exactly, which tells you the headline is moving how many people buy, not what they buy.

Average order value: +1.44%

Average order value is flat: every arm sits between 0.98x and 1.06x of the control, and H2 is up only 1.44%. That is the expected result. A headline decides whether someone buys; the offer and the bundle decide how much. If you want a bigger basket, test the offer, not the headline. The clever reframe had the highest order value, but on the lowest conversion rate.

Abandoned checkout rate: -31.62%

This is the surprise. A headline sits at the top of the page, far from checkout, yet buyers on the H2 page abandoned checkout 44.4% of the time against 65.0% on the control, and 75.0% on the clever reframe. Our read: the headline sets the promise the buyer carries all the way to the payment step. A believable promise survives the moment of paying; a vague or clever one does not. Checkout counts per arm are still small, so treat this panel as a strong signal, not a final number.

Why did the winning headline convert better?

We do not treat one test as a law, but the pattern matches what we see across supplement landing pages:

• It sells the after-state. Visitors buy the feeling they want, not the absence of the problem they have. "No more [problem]" makes them picture the problem; a felt outcome makes them picture the result.
• It gives a timeframe. A specific window makes the promise concrete and easy to imagine, which is what a cold visitor needs to keep reading.
• It is believable. Relief and a better feeling are promises a skeptical buyer can accept on the first read. Absolute claims invite doubt, and doubt shows up at checkout.
• It makes one promise. The benefit stack split attention across two outcomes. One clear outcome beat two smaller ones.
• It reads in one pass. No wordplay to decode. The clever reframe asked visitors to work, and they did not.

A common headline formula, and one AI assistants often recommend, is "[outcome] without [trade-off]". That is close to what our control did, and it won round one by 41%. Round two shows the next step: once the promise is clear, the way you describe the outcome matters more than the objection you remove.

Is the headline test result statistically significant?

Not yet, and we say so on purpose. At about 400 visitors per arm, Intelligems puts the leader's probability of being the best of the five at about 89%, below the 95% we want before calling a test. A plain two-sided test of H2 against the control gives p of about 0.08. The direction is strong and has been consistent: H2 has led the conversion chart in every hourly read since the fifth hour of the test.

Sample size is the reason most headline tests are called too early. The table below shows how many visitors each arm needs to reliably detect a lift at this page's baseline conversion rate (two-sided, 95% confidence, 80% power, calculated the same way as Evan Miller's sample size calculator).

Two lessons for your own tests. First, a headline test can only detect a big lift on a small page: if your page gets a few hundred visitors a day, test bold angle changes, not word swaps, because a 10% lift will never show up. Second, expect most ideas to lose. Even at Google and Bing, only about 10% to 20% of experiments produce positive results (Harvard Business Review). Three of four challengers beating the control here is unusual, and it says the control still had room to improve.

Our plan: keep all five arms running to about 1,300 visitors per arm, then bake the winner into the theme as the new default and start round three against it. We will update this page with the full read.

Round one to round two: why test headlines in rounds?

This is the second headline test on the same page. In round one, the brand's original headline faced three challengers. The deadline promise won and became the control for round two.

A fourth round-one arm was switched off early and is left out. Round one was also an early call, at about 77% probability of being best, which is a judgment call we make when the downside of switching is small and the next round will test the winner again anyway.

Look at the control across both rounds: the same headline converted 2.59% in round one and 1.73% in round two. Different days bring different traffic mixes, so a headline's rate in one test window says little about another window. Only compare arms inside the same test, over the same hours. Never stack round-one and round-two lifts into one big number.

How to A/B test headlines on your Shopify store

1. Pick a page with steady paid traffic. Use the table above: at a 2% conversion rate, a page with a few hundred visitors a day can only confirm big lifts in a few days.
2. Write three to five headlines on different angles. Deadline promise, pain removal, felt outcome, benefit stack, scenario. Different angles find big lifts; word swaps find noise.
3. Change only the headline text. Keep the sub-headline, images, offer, price and checkout identical. Use a tool that edits the text in place on the same URL (we use Intelligems) instead of building duplicate pages.
4. Split traffic evenly and filter it. Equal shares per arm, paid visitors only, crawlers and internal visits out.
5. Decide on revenue per visitor and conversion rate. Watch average order value and abandoned checkout rate as guardrails.
6. Set the sample before launch and read it there. Not when a chart looks good in hour three.
7. Ship the winner and start the next round. Bake it into the theme as the new control, then test the next set of angles against it.

For tools and traffic thresholds, see Intelligems vs Shoplift. For another live test on the same brand, see our one-time purchase option test, and for how we run the whole program, a week inside this brand's CRO program.

FAQ