Open rate is the wrong metric to optimise

Open rate is the first number on almost every outbound dashboard, including ours. It is also the number you should make the fewest decisions with. Not because tracking is broken, but because what it measures stopped being "a person read this" some years ago.

What an open actually is

An open is a request for a 1×1 transparent image embedded in the message. When the image loads, the request reaches the sender and is recorded. That is the entire mechanism, and every one of its failure modes follows from it.

• Apple Mail Privacy Protection, on by default since iOS 15, prefetches that image whether or not anyone reads the message. Those opens are a machine, and they arrive promptly and uniformly.
• Corporate security gateways fetch every URL in an inbound message, including the pixel, to scan it. Same effect.
• Recipients with images off never register an open at all, however carefully they read.

So the number is inflated by automated fetches and deflated by blocked images simultaneously, in proportions that vary by audience and are invisible to you. It is not noise you can average out — it is a bias whose size you cannot measure.

Why this ruins A/B tests specifically

Averages survive a bit of noise. Comparisons do not survive a bias that differs between the things being compared. Test two subject lines and the winner may simply have been sent to a segment with more Apple Mail users. Nothing about the copy needs to differ for the numbers to.

There is also a sample size problem that predates any of this. To detect a 2-point difference in a rate near 40% with any confidence, you need thousands of sends per variant. Most outbound tests are run on a few hundred and called after a day, at which point you are reading a difference smaller than the noise. Declaring a winner there does not just fail to help — it bakes a coin flip into your template library and you carry it forward for months.

What to steer by instead

Prefer metrics that require a human to do something deliberate, and that sit closer to the outcome you actually want.

1. 1Reply rate. Someone typed something. Expensive to fake, impossible to prefetch, and it is the thing you wanted. The only real drawback is that it is a smaller number, so it needs more volume to read — which is a reason to run fewer, larger tests, not to fall back on opens.
2. 2Positive reply rate. Separate "interested" from "unsubscribe me". A variant that lifts total replies by annoying people is a loss you will otherwise book as a win.
3. 3Meetings booked, or whatever your actual conversion is. Fewest data points, most signal. Worth tracking even when the volume is too low to test on, because it is the only number that tells you the campaign worked.
4. 4Bounce and complaint rate, as guardrails rather than goals. These do not tell you a variant is better; they tell you to stop.

So is the pixel worth anything?

In aggregate and over time, yes. A large drop in open rate across every campaign at once is a real signal — usually that you have started landing in spam. It is useful as a smoke alarm precisely because you are watching a change in a broken measure rather than trusting its absolute value. Just do not decide which subject line to keep with it.

We report opens in OutreachPro because that trend is worth having, and because leaving it out would just mean you calculate it elsewhere. We put replies next to it because that is the column to actually read.