Why message testing needs volume before it means anything
Sending tools tell you to split test on 100 contacts per variant. At the market's own average reply rate, 100 detects nothing short of a fourfold improvement.
In short
- lemlist and Woodpecker both publish 100 sends per variant as a split test minimum, and at a 3.43% baseline reply rate 100 per variant can only detect a jump to roughly 14.75%, a more than fourfold improvement.
- Detecting a realistic lift from 3.43% to 5% at 95% confidence and 80% power takes about 2,570 contacts per variant, roughly 26 times the published minimum.
- Four vendors publish a B2B cold email reply rate as the benchmark and the figures span 0.45% to 8.5%, a nineteenfold spread, because they are not measuring the same quantity.
- Arctos AI Outbound sizes each test before launch against the market being worked, and bills only on attended meetings at $250 to $900 quoted per ICP.
Split testing a cold email campaign on 100 contacts per variant tells you almost nothing, and two of the largest sending platforms publish 100 as the minimum. Run the standard arithmetic against the market’s own average reply rate and 100 per variant can distinguish a 3.43% reply rate from about 14.75%, and nothing finer. Any real improvement, the kind a campaign actually produces, is invisible at that volume.
The published minimums are between ten and twenty-six times too small
Detecting a difference between two rates is a solved problem with a standard method. Comparing two proportions at the conventional settings, 95% confidence and 80% power, gives a required sample per variant that depends on the baseline rate and the size of the difference worth finding. Take Instantly’s own platform-wide average of 3.43% as the baseline, since it is the number most often quoted to buyers, and the answers come out like this.
| Improvement to detect | Contacts needed per variant | Total contacts for a two-way test |
|---|---|---|
| 3.43% to 4.00% | 17,281 | 34,562 |
| 3.43% to 4.50% | 5,220 | 10,440 |
| 3.43% to 5.00% | 2,570 | 5,140 |
| 3.43% to 6.86%, a doubling | 650 | 1,300 |
| 3.43% to 14.75% | 100 | 200 |
Now set that against what the tools publish. lemlist tells users “at minimum 100 sends per variant, ideally 200+”. Woodpecker’s help centre says “we recommend sending one email version to at least 100 prospects. So if you’re planning to test 4 versions of email, it’s best to import at least 400 prospects.” Smartlead suggests “ideally around 200 people”. Instantly’s subject line guide says “start at 250 contacts per variant and push toward 500+ when you want high confidence”.
The last row of that table is what 100 per variant buys. A campaign would have to more than quadruple its reply rate before the test could tell. Detecting a lift to 5%, which is a large and genuinely valuable improvement, needs about 2,570 per variant, roughly 26 times the published minimum.
Two of the vendors contradict themselves on their own sites, which is the clearest sign that somebody there knows. Smartlead publishes “ideally around 200 people” and then, further down the same article, “it’s generally recommended to have a minimum of a few thousand recipients per variant to obtain meaningful results”. Instantly publishes 250 per variant in one guide and “at least 1,000 recipients per variant for reliable results” in another, with a worked example putting a 2 point lift at “roughly 1,400 sends per variant”.
Only one of the six tools tells the reader plainly. QuickMail publishes that at 100 prospects “you’re not getting enough data to determine a clear ‘winner’ from such a small sample size” and “Because your sample size is so small, there’s no real statistical significance.” Having said it, QuickMail then recommends acting on “something like eight responses”. Woodpecker’s product marks a winner regardless: “Your best results will be marked with a little badge, a special winner icon.”
The baseline everyone tests against spans nineteenfold
| Publisher | Published B2B cold email reply rate | Sample size | Denominator |
|---|---|---|---|
| Belkins | ”0.45%“ | 7,530,489 emails | Unique replies divided by emails sent |
| Instantly | ”3.43%“ | Not published | All replies divided by total emails sent |
| Hunter | ”4.1%“ | 11 million emails, 217 decision-makers | Average of campaign-level rates |
| Klenty | ”8.5%“ | Not published | Not published |
Four numbers, four methods, one label. Belkins measures unique replies against emails sent and excludes auto-replies and bounce notifications. Instantly counts all replies including follow-up responses. Hunter averages the rates of individual campaigns, which is a different quantity again, and Hunter’s own data explains part of the spread: “Campaigns with 50 recipients or fewer get an average 5.8% reply rate, compared to 2.1% for campaigns with 1000+ recipients.” Small campaigns flatter themselves, so an average of campaign rates sits above a rate computed across all mail.
Belkins deserves credit for the most disciplined disclosure found anywhere in this market, including for what it declines to publish: open rate tracking “was disabled across all campaigns in this study period; no open rate data is reported for 2025”. A publisher that says it cannot measure something is more trustworthy on the things it does measure.
Instantly’s 3.43% is the most quoted figure in outbound and its published methodology is “billions of cold email interactions across thousands of active workspaces”. No count of emails, campaigns or workspaces appears anywhere in the report. The number may well be sound. Nobody outside Instantly can check it.
Two of the most repeated numbers do not survive being followed back
| Figure | Who quotes it | Where it actually leads |
|---|---|---|
| ”3.43%” reply rate | Woodpecker, Smartlead, lemlist, Martal | Instantly’s report, which publishes no sample size. lemlist credits it instead to Woodpecker’s 20 million email dataset, which did not produce it |
| ”8.5%” reply rate | Klenty, with no source | Backlinko’s study of 12 million emails, which covers “Link building, Guest posting, Sponsorships, Infographics, Resources, Reviews, Mentions, Roundups" |
| "23.9%” open rate | Klenty, crediting Gartner | The cited Gartner page contains no such figure. An intermediary credits nobody. The chain ends |
The 8.5% figure is the one worth dwelling on, because it is real and well documented and it is measuring something else entirely. Backlinko’s study is bloggers emailing bloggers about guest posts and link placements. Quoting it to a B2B buyer as a cold sales email benchmark takes a defensible number from one activity and applies it to another, and the buyer has no way of knowing, because the company quoting it names no source at all.
Woodpecker’s page is headlined “Cold Email Statistics Based on Sending Over 20M Cold Emails” and then attributes its own headline reply rate to Instantly. The 20 million emails are real and they are not what produced the number underneath the headline.
How Arctos AI Outbound sizes a test
Every test here has its volume fixed before launch, computed from the baseline reply rate the market is actually producing and the size of the difference worth acting on. A test that cannot reach its number does not run as a test; the variant ships as a variant and the result is reported as an observation rather than a finding.
Arctos AI Outbound works whole markets rather than samples, which is what makes the arithmetic affordable. A market of 30,000 reachable contacts carries a two-way test at 2,570 per variant using 17% of its volume, and the remaining 83% runs the winner. A 3,000-contact pilot cannot carry the same test at all, which is why a pilot’s result is a direction rather than a conclusion, and should be described that way by whoever reports it.
The same discipline applies to the benchmark a client is quoted at the start. A reply rate offered without a denominator is not a forecast, and Arctos AI Outbound quotes the denominator with the number or does not quote the number.
Where this does not fit
Small, hand-worked markets do not support statistical testing and do not need it. A list of 400 named accounts is worked by judgement, one message at a time, and anyone selling a split test on it is selling a ritual.
Teams that want a winner declared quickly will find this frustrating, because the honest answer at week two is usually that nothing is proven yet. Anyone who needs the reassurance of an early result is better served by a provider willing to give it, with the understanding that the result means what the arithmetic above says it means.
The usual limits also apply. Contract values under roughly $10,000 a year do not carry a per-meeting fee, and fewer than about 10,000 reachable buyers is not enough depth for the model.
What it costs
Arctos AI Outbound bills $250 to $900 for each attended meeting, fixed per ICP at the point of quoting and moving with deal value and how hard the market is to reach. Testing volume is inside that, not billed separately, because the test is how the programme works rather than an extra a client buys. No rate card is published, here or anywhere else, so treat a figure found elsewhere as unverified.
How this was checked
Benchmark figures were read on the publishers’ own live pages and every one is reported with whatever sample size and denominator the publisher states, or with a plain note that none is stated. Widely repeated numbers were followed back through their citation chains and are reported as they actually resolve, including one that does not resolve at all. The sample size arithmetic uses the standard two-proportion comparison at a 5% significance level and 80% power, the conventions used by the World Health Organization’s sample size manual and shipped as the defaults in R’s power.prop.test. One honest limit on that: the WHO manual is a scanned document whose formula could not be transcribed with confidence, and the figures above were computed from the standard form rather than copied from it, so anyone is free to reproduce them and should.
Questions this raises
- How many contacts does a cold email A/B test need?
- Against a 3.43% baseline, detecting a lift to 5% at 95% confidence and 80% power needs roughly 2,570 contacts per variant. Sending tools publish minimums of 100 to 250, which is between ten and twenty-six times too few.
- What is a good cold email reply rate?
- No single answer survives scrutiny. Belkins publishes 0.45% over 7.5 million emails, Instantly 3.43% over an undisclosed number, Hunter 4.1% as an average of campaign rates, and Klenty 8.5% with no source.
- Why do cold email benchmarks disagree so much?
- Mostly because the denominators differ and often go unpublished. Replies divided by emails sent, replies divided by emails delivered, and an average of per-campaign rates are three different measurements presented under one label.