Campaign testing: multivariate (or A/B)
Email marketing isn't a one-size-fits-all practice. Trends shift, strategies evolve, and subscriber behavior constantly changes. That’s why testing different elements in your campaigns—like subject lines, layouts, or imagery—can be so valuable. Running consistent tests helps you better understand what resonates with your audience and allows you to make data-informed decisions that improve your overall email performance.
When to Test
Recurring campaigns, such as monthly newsletters or promotional series, are ideal candidates for testing. These emails typically have a performance history, making it easier to spot trends and measure the impact of changes. If you notice metrics like open or click rates starting to plateau, it may be time to try something new. Use testing to determine what updates can reignite engagement.
What to Test
Once you’ve identified a campaign to test, decide which elements to focus on. Delivra’s Multivariate Testing tool makes it easy to compare combinations of subject lines, sender names, and email designs. You can test up to 8 total combinations in a single campaign.
For example, you might test:
-
2 subject lines × 2 email designs = 4 total combinations
-
2 subject lines × 2 sender names × 2 email designs = 8 total combinations
Common testing options include:
-
Subject Line Variations
Try emojis, different punctuation, personalization, or changes in tone to see what leads to higher open rates. -
Sender Name Alternatives
Compare “Your Company Name” vs. a real person’s name to test which drives more opens. -
Design and Layout Options
Experiment with plain-text vs. designed emails, image placement, CTA buttons, or different template structures to boost clicks and engagement.
How to Run a Test
When setting up a test, you’ll need to determine how to divide your audience. There are two popular methods:
-
A/B Split (50/50)
Send version A to half your list and version B to the other half. This is quick to set up and gives you instant comparison data, but all contacts receive different content—no “winner” is sent afterward. -
Test Group + Winner Send
Send multiple versions to a smaller portion of your audience, then deliver the winning version to the remaining majority. This ensures most of your audience receives the highest-performing message, but it requires a large enough test group and adequate time to gather results. Plan to let your test run for several hours—ideally 24—to ensure enough engagement data is collected.
Best Practices for Effective Testing
To get the most out of your email tests, keep these tips in mind:
-
Set a Clear Goal
Know what you're trying to achieve. Whether it's boosting open rates to 25% or increasing click-throughs, define your success metrics upfront. -
Always Include a Control
One version should be your original email format. This gives you a reliable point of comparison for every test you run. -
Use a Large Enough Test Group
A bigger sample size reduces random error and helps ensure your results are statistically significant. -
Test Regularly
Even small, continuous tests can lead to major improvements over time. Build testing into your regular campaign process. -
Let the Data Lead
It can be tempting to follow instinct, but trust the numbers. Your audience’s behavior offers valuable insight—don’t ignore it.
Practical Use Case
Let’s say your monthly newsletter used to have a 28% open rate, but it's recently dropped to 20%. You decide to run a multivariate test with two new subject lines and two alternate designs. The test runs with four unique combinations. After analyzing the results, you discover that a specific subject line paired with a simplified layout performs the best. You apply that combination to your next send and see a 30% open rate—an immediate improvement.
Getting Started
If multivariate testing is new for your team, begin with one campaign. Focus on testing one element at a time and optimize based on what the data shows. Over time, this approach builds a more effective and audience-responsive email strategy.
Want to dive deeper?
Visit our e-Learning hub to explore tutorials on multivariate testing and other automation tools available in Delivra.
Related Articles
I'm not receiving test emails from my account. What can I do?
While there could be a number of factors causing you not to receive test emails from the system, below are typical scenarios and troubleshooting steps. The test email failed in some way. Invalid From lines, blank subject lines, invalid merge tags in ...A recipient of my email reported that they did not receive my emails, but reports show their emails are successfully sent. What can I do?
Chances are, the receiving email server is accepting the campaign but preventing it from being delivered to the inbox. As long as the recipient is at a private/business domain, they can contact their IT team to white list our sending domain or IP ...Automated welcome & confirmation emails
Our system offers two types of automated campaigns that can be configured to send to new subscribers: welcome & confirmed opt-in. What is an automated welcome or confirmed opt-in email? A welcome or confirmed opt-in email is an automated type of ...Why is the sending of my email campaign delayed?
The Tracking Statistics report allows you to view overall mailing performance, including sending performance and status. The column named Completed % indicates the percentage of completion, or delivery, for your email. There may be times that the ...Why is [SPAM] being listed in the subject line of my campaign?
The display of [SPAM] in the subject lines of some campaigns is being added by the receiving server's spam filter. Typically this occurs due to something the receiving server did not like about the campaign. The IT team at the receiving domain should ...