Open Rates Matter? Lol!

From time immemorial, email campaigns have been designed, executed, and evaluated based on three core metrics, open rates, click rates, and reply rates, which we have accepted as axioms for a long time.

Seldom do we question if these metrics still make sense given the recent advancements in technology to help us both track activity of users and block trackers that monitor activity of users.

In addition to these technological advancements, we are witnessing a significant push towards enhanced privacy of users, to prevent their data from being used without their consent, albeit on our products.

It is time we question these axioms of open rates, click rates, and reply rates and re-examine the validity of these metrics as well as our ability to accurately and effectively determine and track these metrics.

Let us start with open rate, which is the subject of this black paper. Open rates have historically been a comfort metric for us, because it is the first and easiest metric to move in our favour, or so we think.

Open rate instils a sense of comfort that the email campaign is working, which from a psychological perspective is very important, because it gives us the energy to move forward with wishful thinking.

Click rate help us narrow the funnel of prospects for subsequent actions on relative more expensive channels like LinkedIn, calls, and ads, based on the assumption that prospects who click links have been warmed up.

Reply rate, which is the most important metric, is also the toughest metric to move, which as counter-intuitive as it may sound, it not the actual focus for most email campaigns.

In reality, when we design and execute email campaigns, we are trying to warmup prospects so that when we target them using other channels, the likelihood of getting their attention is higher.

Given the abysmal reply rates in the recent past, most people have started to adopt this outlook towards email campaigns, an outlook wherein emails are a part of an omnichannel strategy, to largely narrow the funnel.

Open rate is not an accurate metric anymore. We must question our ability to identify prospects who have opened our emails. The purpose of this black paper is to prove with an experiment that we cannot track open rate.

Open rate is most commonly calculated using email pixels, which we no longer can track accurately, as will be explained in detail in the subsequent sections of this black paper.

Click rate as a metric cannot be taken at face value either. Most of us know that even SMBs have started to adopt IT security systems that screen emails to prevent phishing and trojans.

We either conveniently forget this fact when we run email campaigns or do not connect the dots about the existence of these tools and the click rates that we witness in our email campaigns.

IT security systems tend to scan every email before it delivers them to the inbox of recipients. As a part of the scanning process, these systems also open links attached to ensure they are safe.

Unfortunately, tools that we use to run email campaigns may not be able to differentiate clicks from such IT security systems from actual user clicks, thereby inflating the click rate of email campaigns.

Subsequent decisions, including retargeting prospects, basis click rates can be significantly incorrect and ineffective, which we do not take into consideration when we build our outbound lead generation strategies.

Reply rate, the toughest metric to move, is the only unquestionable metric exists today, which may also be inaccurate in the future, when bots start responding to emails instead of humans.

However, this is problem for a later date, as seldom do we find people who have adopted bots to respond on their behalf, in an autonomous way, especially to work emails. We hope that it remains this way.

For now, reply rate is the only accurate metric to evaluate the performance of an email campaign we strongly believe that it must be the only north-star metric or NSM for all email campaigns.

As mentioned above, the focus of this black paper is to prove that we can no longer accurately measure open rate and to do so using a simple yet deterministic experiment as detailed below.

Separate black papers will be authored about our inability to track click rate and the implications of the same, along with our opinions on effective corrective actions for outbound lead generation strategies.

Email pixels are the most common method used to track email open rates as mentioned above. Pixel is a small, transparent 1×1 pixel image that is embedded in the HTML code of an email message.

When a recipient opens the email, the pixel is automatically loaded, which sends a signal back to the server, which serves as an indication of the email being opened by the respective recipient.

The more traditional method of tracking open rates is using read receipts, wherein a recipient is explicitly asked for their consent to send an alert to the email sender stating that they have opened the email.

While this is a more reliable method to determine email open rates, there are certain apparent and latent issues with this as mentioned below:

• Read receipts may not be supported by all email clients
• People may disable the feature even if email clients support it
• Recipients generally tend to avoid sending confirmation to senders

Email pixels overcome these shortcomings of read receipts as they can be loaded on almost every email client and they do not require explicit action from recipients to share information about email open rates.

Gmail tends to preload email pixel which fires the pixel before the email is actually opened, leading to inflated open rates, which is difficult to correct or adjust and therefore misleading.

Microsoft tends to block images, irrespective of their size in emails by default, which prevents email pixels from loading in their email client, thereby blocking information about email opens.

Other challenges include region specific compliance requirements like GDPR, which require user consent before using tracking pixels to gather information about email opens, which pose legal challenges.

As a safeguard, we suspect certain software may not embed email pixels for emails that are being sent to regions like Europe, because user consent is seldom obtained in opt-out emails like lead generation emails.

If people who use email pixel truly understand these problems and related shortfalls, they can better manage their expectation regarding reports about email open rates, which is a rare occurrence.

Unfortunately, people who manage email campaigns are so biased and excited by email open rates that their subsequent set of actions are largely based on this metric, which can be significantly incorrect.

We have witnessed these actions based on email open rates by marketing, inside sales, business development and even sales teams:

• LinkedIn messages to prospects who have opened emails
• Cold calls to prospects who have opened multiple emails
• Ads, where possible, to retarget people who have opened emails
• Nurture campaigns for prospects who have opened emails
• Expensive data enrichment for prospects who have opened emails

As a result, significant effort and capital tends to be wasted on prospects who are considered warm, because of incorrect and inaccurate inferences that they opened emails, which is an indication of interest.

An even bigger problem lies in the inference that when a prospect opens an email, it is an expression of interest in the topic that is core to the email, which from a common-sense standpoint, is also incorrect.

Logic dictates that if a prospect is really interested in the email that is sent to them, they will act on it, either by replying to the email or by clicking a link that is in the email or by searching online about the sender.

The only accurate and unquestionable expression of interest is that of a reply from the prospect, which as mentioned earlier in this paper, must be the only north-star metric or NSM that sellers must follow.

To prove what we have stated above, about open rates being unreliable, let us formulate an experiment wherein we design and execute an email campaign using a popular email campaign tool.

For the purpose of this experiment, let us consider the following:

The expected outcome from this experiment is that open rate from the Gmail ID may be captured accurately on E-Tool-X but the open rate from the Microsoft ID will not be captured accurately on E-Tool-X.

Three mailboxes are in scope for this campaign, i.e., the seller’s mailbox (growth@lifex.services), recipient’s Gmail inbox (np.lifex@gmail.com) and recipient’s Outlook inbox (marketing@getreplies.ai).

Mailbox: growth@lifex.services

Mailbox: marketing@getreplies.ai

Mailbox: np.lifex@gmail.com

Note that neither of the recipient mailboxes i.e., np.lifex@gmail.com nor marketing@getreplies.ai have an email from the seller’s mailbox address i.e., growth@lifex.services as shown in the screenshots.

The seller’s mailbox i.e., growth@lifex.services has been added to the list of mailboxes in E-Tool-X and the setup is complete as is shown by the setup progress bar which stands at 100% for this mailbox.

To preserve confidentiality of information that is part of this account, certain elements have been blacked out, including but not limited to email IDs, account name and details of past campaigns.

You can also clearly see that no emails have been sent from using this mailbox today as the daily limit indicators reads 0 out of 50 and deliverability score has no information as no emails were sent.

The single step campaign will have the subject line as “Test” and the body of the message will read “Experiment to test email open rates”. It will be sent to both the recipient IDs mentioned above.

A CSV file named “test_data” was created which had contact details of the recipients as mentioned above. The snapshot of the CSV file is also shown below for your reference.

Check both the Gmail & Microsoft mailboxes to ensure that the emails have been delivered and that they reach the inbox and not spam folder. In case of spam delivery, mark the email as not-spam.

The image above shows that E-Tool-X has delivered the email. Now we must confirm the same using the individual mailboxes and ensure that the email was delivered. The account name of the sender will be blacked out.

The images above show that both, the Gmail and the Microsoft recipient IDs have received the emails. The next step is to open the emails and check if E-Tool-X is able to accurately detect the opens.

We will now open the email from Gmail and then check after five minutes if E-Tool-X is able to detect the email open in Gmail. Our best guess at this point is E-Tool-X will be able to detect this open.  

As expected, E-Tool-X is able to detect the open in the Gmail mailbox. 

We will now repeat step-6 but for the Microsoft mailbox which we know will not render images, even if it is just a pixel, without explicit action from the user, therefore preventing detection of open rate.

Seven minutes later, E-Tool-X is unable to detect the open in the Microsoft mailbox. Rest assured, even a day later, the result will be the same.

The experiment clearly proves that email open rate is not detectable when the recipient has a Microsoft mailbox, but the open rate is detectable when the recipient has a Gmail mailbox.

We are NOT to conclude that if recipients have Microsoft mailboxes, the open rates can never be determined, NOR are we to conclude that that if the recipients have Gmail inboxes, open rates can be determined.

The purpose of this black paper was to prove that open rates are unreliable today given the shortcomings of the pixel approach to track if emails are opened as explained earlier, which we have successfully done.  

Depending on firewall and mailbox settings of an enterprise, even Gmail mailboxes may not render pixels and Microsoft mailboxes may render pixels by default. Either way, it is beyond the seller’s control.

It is not unreasonable to assume an even distribution of work email IDs between Gmail and Microsoft. If the reality is skewed, it most likely will be skewed in the favour of Microsoft i.e., more Microsoft IDs.

Therefore, if we send 100 emails to prospects in small, medium, and large enterprises, we are likely to reach approximately 50 Microsoft and 50 Gmail inboxes, assuming no other email provider.

Most enterprises will either not change the default image rendering settings. Even if they do, they will change it to prevent images from being displayed by default without explicit user action.

Under these reasonable assumptions, if an email campaign gets an open rate which is beyond 50%, it is worth asking if the number is reliable and if decisions can be made using the same.

Please note that the purpose of this black paper is to highlight technical challenges associated with tracking email open rates in a reliable manner and not to critique any existing email campaign software.

We are building a Gen AI tool for personalized email campaigns in which we have eliminated the metric called open rate. We have included click rate, but with necessary disclaimers on the efficacy of the metric.