In marketing, there’s a simple rule. You have to test what you do to learn. Learning to test what you do, and finding out what works best is precisely the tool you need to advance your knowledge and reach the next frontier. Here’s how to begin.
When I started with conversion rate optimization, I remember it was a bit scary.
I had to learn what to do and when to do it, and it didn’t end there:
- Information was all over the place back then, and you had to keep focus.
- Learning about a testing methodology is not exciting, to say the least, although once I got It under my belt, it blew my mind every single time.
- And those who test what they do might fail from time to time. And I did, a lot.
To overcome my fears, I had to adapt, and here’s how I did it.thenticated emails”
I love Astrophysics, and it’s only natural for me to associate unknowns with the big unknown.
Why is it natural?
Because as it turns out when you want to learn something new, you need to associate it with something you already know. That’s how learning works.
My passion fueled my learning, and through Astrophysics, I learned conversion rate optimization.
Suddenly, from a thought, I created a universe in my head and started to jot it down.
It was my big bang.
That universe expanded rapidly to objectives, emails, banners, landing pages, copy, and just about everything else, much like the galaxies, solar systems, and planets.
I mapped everything and created hypotheses and heuristics and started to design my first email marketing tests.
When I mapped out all the elements in my email marketing universe, I used a top-down approach, let’s see how it looks on a high level from conceptualization to CTA:
- Finding the campaign that will help me achieve my business objective was my big bang.
- My galaxies are the personas, user journeys, funnels, and their varying stages.
- My solar systems are emails, landing pages, and ad platforms.
- My planets are the designs, wireframes, UIs, and UXs.
- And the elemental table is the copy, CTAs, and button size/location/colors.
Now that we know how the CRO universe looks like, it’s time to develop a framework that will allow you to contain and test it.
The complete
deliverability
handbook
Read the most significant, most organized volume of information written about email deliverability.
Before we dive into email marketing specific tests, we need to understand the testing methodology as a whole and its impact on our marketing efforts.
Why should you run conversion rate optimization for email marketing?
Email marketing today is a holistic practice. It’s not just subject lines and email copy. You have your:
- Target audience.
- Delivering your emails and deliverability.
- The email you’re sending.
- The products you offer.
- The landing pages they lead to.
- The journey of your user or client.
- Nurturing them after the purchase.
- Email list growth, and much more to think about.
When you have one set of variables, actions to perform, and elements to test, the above might look pretty straightforward. But double it by x2, and you’ll get a hot mess.
To avoid that mess, and to create a process that you’re able to scale, follow the below framework and connect your email marketing end to end.
Time to see how everything connects!
via email
By signing up you are agreeing to our Terms of Service
Your data will be handled in accordance with our Privacy Policy
The data collection and research stage
In CRO, our universe is made out of data, and we use that data to achieve our goals.
Data that you collect or own and its analysis is the base from which you make hypotheses (see next step) and draw conclusions from (the step after that).
Which kind of data are we looking for in CRO?
User or client behavior.
If we know what our clients do, we can encourage them to repeat certain actions, or increase the likelihood of certain behaviors.
How to collect and analyze data?
CRO data usually comes in two forms, data that can be counted, measured, and expressed using numbers (quantitative) and descriptive or conceptual data (qualitative).
Whether raw text or numerical values, all data points are important and can provide insights into user or client behavior.
Each type of data can be collected and researched via different tools and methods, for instance:
- Quantitative – survey tools, A/B tests, multivariate tests, hypothesis heuristics, available data (Analytics, CRO maps, user recordings), etc.
- Qualitative – survey tools, usability testing, customer immersion, customer Interviews, sales rep interviews, etc.
You don’t need to employ all of the above tools and techniques, just those that will help you to gather the behavioral data you need.
How much data do you need to make a decision?
The amount of data you have can impact your decision making positively. A large enough data sample will allow you to make decisions with confidence.
Why does confidence matter in testing?
Because data that repeats itself through a wide range of samples is data you can classify as recurring behavior.
Why at the lowest cost? Because collecting data costs money, and you want to calculate your costs until the minimum detectable effect optimally.
I won’t dive into how to calculate sample sizes, minimum detectable effect, and other test results, there are great posts which already covered that, and ready-made calculators you should check out.
A word of advice, make sure that you calculate your test metrics before you set off and plan when to introduce new tests once you acquired results that you’re confident about.
The Email Marketing Activity Book for Kids
The CRO test execution methodology stage
Now that you know what type of data you’re looking for, how to collect and analyze it, and the size of the required sample size, you can dive into your CRO execution strategy.
But before you create tests, you’ll need to determine what to test first.
How to prioritize your CRO tests?
Prioritizing tests is tricky.
Our natural tendency is to try and test everything, implementing it the minute we can, and launching a perfect funnel from the get-go (or so we think).
Perfect is the enemy of done, as you already know.
That’s why you have to choose first, what matters the most.
There are effective ways to prioritize tests that you can quickly wrap your head around. For instance, you can assign each of the below factors a weight that ranges from 1 to 5:
- Level of impact (1-5): 5
- Probability of impact (1-5): 3
- Implementation difficulty (1-5): 5 (5 here is easy)
- Urgency (1-5): 4
Total: 5 x 3 x 5 x 4 = 300
Bryan Eisenberg, who invented this model, goes into detail about it in this post, but the above illustration should demonstrate it.
If you have a large number of tests, you can take the factors and organize them in a table for quick prioritization like the below example:
How to develop and design a CRO hypothesis and heuristics?
Your CRO hypothesis is a scenario in which you:
- Describe a problem.
- And try to predict a possible solution.
These are based on, and help to further shape, your personas, user journeys, and funnels.
When you create a hypothesis, you really need to dive into the data you own and describe what the suggested heuristics are.
Heuristics are assumptions you use to Identify the factors you can influence. These factors can help increase the probability of action, in our case, an open rate, a click-through, or conversion.
When creating your heuristics, ask yourself questions like::
- Who will this change impact?
- What evidence are these suggestions based on?
- Why do you think it will lead to improvement?
- What are the expected results?
You can dive deeper into creating heuristics here.
After you organize the above hypothesis and jot it down, you’ll be able to plan test variations for the next stage.
How to develop creatives and copy variations?
After the development of your CRO hypothesis and heuristics, you’ll be able to derive the types of creatives and copy variables you require.
The email marketing ecosystem is based on three major “test fields” or areas.
- Wireframes – are the structure of your email or landing page. In them, you create areas in which design elements or text can be inserted and tested.
Your wireframe determines the location, size of level of exposure to different elements in your copy and design. - Copy – when testing your text, you need to focus on your target personas and the stage in which they are in the funnel.
When you create variations of your copy, make sure that they’re aligned with the emotional and behavioral response you want to elicit from your target audience. - Design – is anything from colors to images. Even the box shapes in your wireframes.
Your design needs to wrap everything up and provide a sense of confidence that will increase the likelihood of your target persona to perform the required action.
Keeping your tests focused will allow you to tell with confidence that certain things work, which will lead you to the next steps.
Note: designs are incredibly susceptible to biases, make sure that you always run them via a peer review process.
Remember, the best authors have editors. Someone needs to review your work and to make sure that your test is in line with your hypothesis.
Should you conduct A/B or multivariate tests?
A/B tests are variation tests that compare two variations of elements, A and B, and determine which one performs better via statistical analysis of the same test sample.
For example:
Traffic
⬇
⬋⬊
A B
Multivariate tests are a method in which several elements are tested at the same time, in order to find which element combination performs better.
For example:
Traffic
⬇
⬋ ⬋ ⬊ ⬊
Title 1 Title 2 Title 1 Title 2
⬇ ⬇ ⬇ ⬇
Image 1 Image 1 Image 2 Image 2
To conduct either test successfully, you’ll have to have in place:
- At least two variations, and the ability to track data.
- A large enough test sample.
- A testing period in which you’ll let the test run uninterrupted.
- And the ability to learn from your success and failure.
The more focused your test is, the more likely you are to learn something from it.
You can only learn something from your test if you’re confident about the results. That’s why when you’re starting with CRO, it’s best to focus on A/B tests and not multivariate tests.
A/B tests provide results with higher confidence because they test two outcomes and not ten.
However, multivariate tests provide results faster, and with a large enough test sample, which in multivariate tests needs to be really large, you’ll be able to gain confidence about the results.
Notable CRO pitfalls to avoid
Test results are subjected to biases, and as mentioned above, in CRO, confidence matters.
That’s why you have to remove factors that might provide you with false or negative positives.
Among those you will find:
Seasonality
In certain periods during the year, there’s more or less likelihood for a test to work.
The same happens in local events like climate events, and global events like the coronavirus outbreak.
HiPPO
This refers to the “highest-paid person’s opinion” – when someone who holds a high position has a preference that is not based on data.
This usually results in them not being happy with the results, even if they suggested the change.
Running insignificant tests
Tests can be small in scale.
In fact, there are a lot of tests like copy or area design changes that result in significant improvements. However, some tests have an insignificant impact on performance.
For instance, there’s a widespread belief that changing the color of a button or font affect your conversion rate. Although they might, this is not what CRO is all about.
Unfocused focus groups and RATS (random acts of testing)
When gathering focus groups, it’s highly recommended to keep them focused.
When uncontrolled, focus groups tend to derail tests with random ideas.
The test scenarios you’ll gather from these groups might sound like they’re worthwhile. Still, unless these are focused on areas that will contribute to the impact of the test according to the hypothesis, they’ll yield irrelevant results.
Biases
Our brain has a limited amount of resources it can activate at any given time.
If we strain our brain, we’ll experience a condition called decision fatigue, which will demolish our ability to reach the right decisions.
In fact, our brain has already a pattern recognition module in place that helps us to avoid decision fatigue. I’m talking about our biases and prejudices.
Biases and prejudice are impulse reactions that trigger a response that will use the least amount of effort from our side. And when we’re not using our brain to think clearly, it will reach decisions that suit our own inner set of beliefs/biases/prejudices.
For a full list of human biases that might derail your ability to reach the right decision, go to this link, you’ll be amazed.
Things that worked for others
This is a trap that almost all marketers fell into at one point.
When someone publishes results from a test they made, we tend to forget that this test was run in its own universe, and was subjected to different rules.
Even if we’re talking about the same product, in the same industry, with the same price, you can’t assume that you’ll get the same results.
Moving back to our universe, let’s explore how you can effectively test your email marketing efforts.
Conversion rate optimization for email marketing – how to find your tests
After our CRO process review, we’re ready to dive in.
In email marketing, each metric has a story to tell about how your campaigns are doing:
- High delivery rates with low click-throughs could mean you need to optimize your copy, design, or reposition your CTA.
- High click-through rates with low conversion rates mean you should probably change your landing page. Or, check if there’s a mis-match between what you promised and what you’re delivering.
- High delivery rates with high unsubscribe mean you probably need to change your content. Maybe add some personalization?
Each scenario can result in a few tests, but it’s important to remember the essentials:
- Find the right test.
- Make sure you have a large enough test sample
- And don’t jump ship when a change in the subject line yields better click-through rates, but somehow hurts your overall conversion rates.
Tests need to follow the hypothesis you set and must be followed through in full.
That’s why it’s important to determine your testing methodology beforehand, and whether you should focus on A/B testing or multivariate tests.
And with that said, it’s time to talk about email marketing CRO test scenarios!
CRO tests for opens
Before anything, check authentication protocols
“Why don’t they open my emails?”
It’s easy to fall to the “it’s the subject line” trap when your recipients are not opening your emails.
There’s a good reason why it’s your first reflex, but there’s also a possibility that the emails you’re sending are not reaching their destination.
This might happen because they bounced back by ISPs, or got thrown into the spam folder.
As you probably remember:
- When your emails don’t arrive to recipients, it might be an email delivery issue.
- When they don’t land in the inbox or the promotional tab, it might be an email deliverability issue.
Naturally, both will impact your engagement rates.
But, while the second scenario deals with where your email lands? The first deals with did It reach the recipient?
Non authenticated emails will not make it to your recipient’s inbox, changing subject lines won’t provide the required bang for your buck.
So before you dive down the testing rabbit hole, first make sure that you appear as a legitimate sender.
By setting up authentication protocols such as SPF, DKIM, DMARC, and BIMI, you signal the ISPs of the world that you are who you say you are – a legitimate/authentic sender.
Once ISPs detect these signals, they are less likely to filter your messages, which often means that you can get through to your recipients’ inbox.
After you set up your authentication protocols, you’re ready to check delivery rates, subject lines, and much more.
Subject line CRO tests
Your subject line is your recipient’s first impression and first impressions matter.
In a sea of emails, your subject line’s structure, elements, and copy can make a difference between an opened and a non-opened email.
Here are a few tests you can run to try and improve your open rates via your subject lines, remember, each vertical, brand, product and even hour of the day can make a difference, so make sure that you read the below recommendations, and test them:
Emoji test
Studies show that adding emojis in the subject line can increase your CTR and open rate. One study exhibited an increase of 93% in CTR, while others showed a more moderate increase of 56% and 20% in open rates.
Your recipient’s inbox is typically filled with text, so interrupting this pattern with emoji is one way to attract more views.
Remember
- More views don’t equal an increase in open rate. The copy and email need to be relevant and personalized (more below).
- Last thing, make sure that the emojis you use are email-compatible.
Personalization test
Personalizing the subject line can increase open rates by 26%.
While using {{first_name}} can be effective, try to think beyond that with your email personalization strategy. Incorporate data points that will catch users’ eyes, such as their location, interests, purchase history, and more.
This ensures the relevancy of the email from the get-go.
Character limit test
You don’t want your subject line to get cut off mid-sentence, right? Or do you?
Although it had been known that character length could influence open rates, and there were even industry standards that had been issued, the truth is that there’s no one truth.
- Some personas think that a subject line that gets cut off in the middle looks bad.
- Some might respond with curiosity, clicking to see what’s written in the hidden part of the subject line.
- While some will be indifferent, you’ll have to test this.
To see how your subject line looks, use email inbox preview tools such as this one to help you optimize the ideal length of your subject lines.
Value proposition test
In an interview with Jeanne Jennings from Email optimization shop, she explained how opening up with a value proposition boosted revenue per email by 15%.
What does it mean for you?
Instead of subject lines that start with something cryptic like “hey” or “thought you might like this,” you’ll need to test subject lines that explain what’s the purpose of the email.
Sure, without the curiosity hook, you might get fewer opens, but those who will open your email, are more likely to interact with it and click through, which leads us to the next tests.
Different Sender/From CRO tests
This is the “From” section, it appears before your email’s subject line.
This small piece of inbox real-estate has quite an impact on your open rates.
In fact, 68% of Americans decide whether or not to open an email based on this section alone.
So keep the noreply@, marketing@ email addresses for your transactional emails.
When it comes to your regular communications, you’ll need to test which sender names result in more opens.
In most cases, familiar names that had built clout over time with their audience take the jackpot, and it’s important to build the right name from the get-go.
The below “From” email addresses are the usual pick for most companies:
- A generic marketing@domain.com email.
- Your CEO’s personal email.
- Company poster boy/girl – using someone from the company that people can identify in your marketing emails.
- A fictional mascot for your company.
See which ones perform best, and continue to improve and test from there.
Preheader text CRO tests
The preheader text is the short copy that comes after the “From” name and subject line. Ideally, it should give your recipients an idea about what your email is about and entice them to read further.
But if your preheader text looks like the one below, then you’re wasting a valuable piece of email real-estate!
———————–
———————–
Below are examples of email preheader text done right. As you’ll see, they do not waste space by displaying an error message or simply reusing the subject line.
Instead, the preheader texts add more details to the subject line and even a quick CTA for them to read the content like the ones below.
———————–
———————–
Remember, test results require confidence.
If you’re asking what confidence means in email preheaders? Then the answer is being able to distinguish between what led to the increased click-through rate:
- Was it because you wrote a great preheader CTA?
- Or was it the subject line?
Give users a curiosity-inducing (but relevant) preview of what your email contains.
You can also tease an incentive or personalize your message, such as mentioning a special offer just for them, based on their search history.
CRO tests for CTRs
Body copy and design CRO tests
Some email marketers say that short emails are more effective and that you should have only a single call-to-action (CTA). Meanwhile, others preach the complete opposite.
Whatever you heard, know that neither is true for all types of campaigns.
There is no single email template that ensures high engagement metrics on every campaign because there is no one type of user, product, vertical, or intent.
For instance:
- A short email may be best for SaaS, but a long email works better for health and coaching.
- A plain text email may work best for some sectors, while an e-commerce email works best with images and HTML formatting.
Usually, three types of tests apply to the body of your email, and these can be split to further tests, for instance:
- Wireframe (the structure)
- Whether to start with the CTA or use a lead-up story?
- Whether to use one-liners or short sentences or large paragraphs?
- Where to place images?
- Where to add the unsubscribe link?
- To use HTML optimized emails? Etc.
- Copy (the message and CTA)
- Whether to tell a story?
- Feature a testimonial?
- Present a statistic?
- Where to offer the value proposition?
- Should you use direct response copy or soft sell CTA?
- Casual or formal language? Etc.
- Design
- Which colors to go with?
- Should you use images or videos?
- Whether to format your email as plain text?
- Font size? Etc.
All your tests, as previously mentioned, should follow your brand’s guidelines, and focus on your target audience. The last thing you want to do is zigzag every second Thursday with a new test.
Personalization CRO tests
Most marketers believe that the best way to increase your open and click-through rates is via personalization.
They are not wrong. Personalization is highly effective in improving various engagement metrics. But which types of personalization tests are right for you?
It depends.
You can manually personalize small lists and even send them via ISPs, or you can use simple ESP tools to personalize first names, etc. at scale.
But what if you want to dive deeper into the realm of personalization?
There are countless ways to give your emails a personal touch using sophisticated email marketing platforms that start from:
- Utilizing dynamic feeds that insert according to data sets relevant information into blocks inside the email.
- Through hyper segmentation that takes behavioral, emotional, and statistical data to offer the right value proposition, at the right time.
- All the way to easy-to-use tools such as NiftyImages and LiveClicker.
We created here a comprehensive guide that will walk you through the different types of personalization and provide you with insights into the types of tests you can run.
Accessibility CRO tests
Over 200 million-plus people around the world have some form of visual impairment.
This means that a sizable part of the population isn’t experiencing your emails the way you intended. Or worse, they are unable to interact with your emails at all.
- There are people out there who can’t read your CTA button because they’re color blind.
- Most can’t read your copy because your font is too small.
- And some suffer from conditions that prevent them from focusing on large chunks of text.When they don’t land in the inbox or the promotiona
The equation is simple, the more focused on accessibility your email will be, the more eyes it will get. That’s why:
- Test font sizes – small fonts are more taxing to read, but bigger fonts might look ugly, you’ll have to test what’s the sweet spot.
- Test clickable links – people immediately assume that something in blue and underlined is a clickable link, so don’t deviate from this format. You can, however, test what happens if you make your clickables bold and in a larger font.
- Test white space liberally – don’t overwhelm your readers with walls of text. If your emails look like a page from a textbook, you are likely to turn your audience off from reading any further. Add enough space and padding in between your content to make it easier to read.
- Test simple, high-contrast color schemes – the goal is to make your text easier to read. So don’t go crazy with your colors. Stick to high contrast duos of dark text against a light background or visa versa so that your text is easily distinguishable.
- Always add correct ALT text for the media that you use – visually impaired people may not be able to see the media that you use. By using ALT tags, or text that describes what your media is about, text-to-speech machines that the visually impaired users can read aloud what your media is about. This is not a test, just do it.
- Add a text-only option – HTML is great for visuals, but for those who find it easier to read plain text, make this option available in your emails.
Platform, responsiveness and mobile CRO tests
What looks good in Gmail can look totally different in Outlook. What’s readable in an iPad may seem odd in a Samsung Note.
Check your list to see what platforms your audience use to read your emails. Then test preview your emails on these different platforms to make sure your design is consistent, and nothing is broken.
Statistics show that up to 77% of email engagement will come from mobile.
To check how your email renders on different mobile displays, use inbox preview tools such as the ones below.
Test SMS with your email campaigns
On average, 90% of SMS messages get opened within 90 seconds, and 98% of all text messages are opened.
This makes SMS a wonderful conversion booster, and you can test if following up with an SMS after your email campaign was sent, translates into more conversions.
Below are some scenarios you can test SMS follow-ups in:
- A last-minute reminder on a sale/event you previously promoted.
- To highlight a specific product or feature mentioned in the email.
- To target specific segments of users who interacted with your email.
For more details, read our guide on how to use SMS to boost your email campaigns.
What’s after conversion rate optimization?
In marketing, there’s a secret that few marketers know and even fewer practice. The more you test, the more you learn!
Like the universe around us, our marketing knowledge should expand with time, and learning how to test what you do, and find out what works best, is precisely the tool you need to reach your final frontier.
What’s after the final frontier you ask? You can look at other ways to optimize your email operation with tools that examine your sender reputation like Google Postmaster. Happy optimization!
Login