• Oct 29, 2026
  • 3 min read

Change One Thing: A Subject-Line A/B Test With 60 Leads per Arm

Two weeks of sending returned one opt-out and no positive replies. The weekly rule says: replies under 1% after about 100 sends → change one thing, cheapest first. The order I wrote down before sending anything:

  1. The subject line
  2. The type of finding the email leads with
  3. The trade
  4. The offer

Why the subject

It’s one field. And the current one has a problem I only saw once the emails were in real inboxes:

Example & Co: 20s+ load time

It’s accurate, and it reads like an automated monitoring alert. People filter those without opening them. The body, a plain sentence about their own site with an offer of a free report, never gets the chance.

The two variants

Subject
A (control){{company}}: {{finding_short}}
B (test)Question about the {{company}} website

B is vaguer, which is the point: it reads like a person wrote it, and the specific finding is still the first sentence of the body.

What stays frozen

  • The body and the follow-up, word for word.
  • The finding rules: same thresholds, same rounding down.
  • The sending setup: same inboxes, same hours, same daily pace.

The batch does use new trades and new cities (there’s no other way to get fresh leads), but both variants draw from the same batch, split 50/50 by the sending tool. A trade effect lands on both arms equally.

Judging a test that’s too small to judge

About 115 leads, so ~57 per arm. At realistic reply rates (1–3%) that’s somewhere between zero and two replies per arm. No significance test will say anything useful about that. So the rule, written down before launch:

  • Any positive reply, or a clear gap in human replies (3+ vs 0), picks the subject for the next batch.
  • If both arms stay at zero, the subject wasn’t the problem. The next change is the finding or the offer, and the subject goes back to whichever is easier to read.
  • Bounces or opt-outs above 3% stop the test, whatever the replies say.

This isn’t science. It’s a pre-committed decision rule, which is the part that matters when the numbers are small: it stops me from reading a story into one reply after the fact.

What I’ll record

Per variant: emails sent, human replies (auto-replies excluded), positive replies, opt-outs, bounces. One catch: the lead export doesn’t say which variant each lead received. The tool’s A/B report does, so I save it alongside the export every Sunday.

Results in a later post, whichever way they go.

Takeaways

  • Write the order of changes down before you need it.
  • Freeze everything except the variable you’re testing.
  • Small sample? Pre-commit a decision rule instead of pretending to do statistics.
  • Check what your export can’t tell you before the test starts.