Questions & Answers from December’s A/B Testing Webinar

In December’s webinar “Creating a Culture of Testing: 30 Tests in 30 Minutes,” speakers Huw Ap Dewi of Wrexham County Borough Council in Wales and Madeline O’Phelan with GovDelivery discussed how to use testing tactics like A/B testing to improve digital metrics that lead to stronger outcomes. The audience submitted a number of great questions, and without enough time to respond during the webinar, we’ve posted the responses here.

Question: On “humanizing” the communications to your audience, how did you convince higher-ups that testing a more “fun” email is a good idea?     

Huw Ap Dewi: We’re still working on this one! We’ve been working hard on the digital agenda and trying to explain that the old ways of communicating information is possibly not always the best way to do it digitally. We’re aware that memes and GIFs are very popular at the moment and that this is something we want to tap into. The benefit of using the A/B test is that you can choose the number of people who receive the new style/meme bulletin. What we’re going to do is set up an A/B test and we can then use the results to support us when presenting to the higher management team.

Question: I noticed that Huw used “Drip” for the campaign. What does that mean?  

Huw Ap Dewi: The drip campaign option is the option to schedule multiple bulletins within one campaign. This is the tool we use to send out our bin reminders and the engagement and re-engagement campaigns. You can then report on individual bulletins or the campaign as a whole.

You have multiple other options within the drip campaign including the ability to schedule successive bulletins based on the engagement/reaction to previous bulletins within the campaign.

You could also use it if you have a series of bulletins that you want to send over a period of time. An example of this is that we use the drip campaign with our school admissions team. They will know the dates for when they want to promote closing and opening dates for applications at the start of the year. They can then schedule all the bulletins they want to send at the start of the year and then leave it to automatically send throughout the year.

Question: Is the testing best done by using one version for a while, then the other? OR, splitting up your subscribers, where each group would receive a different version?

Madeline O’Phelan: A/B testing is best used when you can test a slight variance in your digital message against a control. The goal is to continuously improve on your digital messages, and to send one version to your a portion of your audience (A) and the second to the other portion of your audience (B) to see what message performs the best. Ongoing, the hope is to build a tested and true message that has data to confirm its success.

Question: What metrics should you look at to measure text formatting and text layout?

Madeline O’Phelan: When testing for text formatting and layout, watch for click rate – especially for the hyperlinks that are farther down in your message. This will tell you if your readers are following along in the body of your message.

Question: How often should you be testing something like layout to build new templates to reuse regularly?

Madeline O’Phelan: We would recommend reassessing your templates at least once a year, if not more. Giving your templates a new look and even testing basics like the color of your header can be a great project to start off the New Year.

Question: Any tips for tracking the results of our tests so we find the best blend?

Madeline O’Phelan: Success of your digital message is determined on a case-by-case basis, so your metrics may look different for every message. However, our latest Benchmark Report has a full list of the five metrics that matter most to your digital engagement strategy and can help provide a place to start.

Question: How much time do you give to gather results of a/b testing in 10/10/80 scenario before declaring winner and then sending to the 80%?

Madeline O’Phelan: Typically, you’ll be able to see the “declared winner” of the A/B test very soon after the initial send. However, a best practice is to let the testing time period last at least 4 hours, ensuring that your audience has had a chance to read your digital message.

Question: Are there resources available within GovDelivery which provide best practices for engagement and outreach which are industry specific? I would like to know if there is a law enforcement agency that has developed a good model for using GovDelivery resources.

Madeline O’Phelan: Yes, we have plenty of resources including success stories available by vertical that you can view on our website. To find resources specific to public safety, click here.

Do you have more questions about A/B testing or creating a culture of testing at your organization? Contact us at info@govdelivery.com and we’ll connect with you.