Can a focus group improve email marketing? After all email is rich in metrics already.
I talked to Gretchen Scheiman from L5 Direct and discovered that if you ask the right focus group questions the answer is a clear yes – there is a very good point to email focus group research.
Gretchen is a smart marketer whose experience includes large and small focus group research. In the following interview she shared her experience with me and explains the why and how of email marketing focus groups.
Email is rich in metrics and just about all email tools support A/B testing. Why do a focus group with so much data already?
A/B testing is very important, and all marketers should have a strategy which guides the tests they focus on every time they pull together an email. But there are things that you cannot learn through A/B testing. For example, it’s possible that while you’re busy learning whether your audience will respond better to choice A or choice B, they’d be much happier with choice C. A well-constructed focus group has the potential to tease out this kind of information, which can then inform your future tests.
What have been some of the most surprising and interesting findings from your focus group work?
My favorite was the time when we spoke to small business owners, and asked them which was their preferred channel for marketing communications. They universally said “direct mail”, which matched our survey results. (This was years ago). But in a focus group, we could probe a bit, so we asked them who handled the direct mail. They admitted that our lovely direct mail pieces that they “preferred” went from the mail sorter to their administrative assistant to the trash bin. Really, they just preferred not seeing them.
We then thought to ask, which channel are you most likely to respond to? Their reluctant answer: email. It is a great story to show that what people say isn’t necessarily what they do, which is why it’s so important to test anything you learn from focus groups.
Most of the time, the findings aren’t surprising. That’s okay, sometimes just getting some answers that match your expectations is helpful and can save you a lot of time in testing. Or guide you to new tests that drive results faster and better.
We had a client that was under a lot of pressure to keep a “share to social” email in their welcome series even though the email wasn’t getting a lot of clicks. The responses from the large-scale focus group to this email were universally negative. The client wasn’t surprised, but now they had ammunition to take back and convince their stakeholders that the email needed to be pulled from the line-up.
Who should be invited to a focus group? How can a brand ensure the right people are included?
Any research should start with the big question: what do you want to learn from a focus group? That’s going to guide your audience selection and the questions you ask. From there, you want to focus on the criteria that might be important to your analysis and your questions. For example, if it is important to your business to understand the perceptions of men vs women, or millennials vs gen x-ers, you’ll need to have that information on your recruiting file, and you’ll need to set quotas for the information so that you make sure that the group you end up with looks like your audience. A good research partner will have experience setting this up, and they will help you work through the details.
Have you any recommendations on the types of focus group questions to ask?
The most important thing is to not ask yes or no questions. You want to listen to your focus group’s opinions and ideas, so you need to structure your questions to generate a response beyond “yes” or “no”.
“Why” questions tend to work well, especially as follow-ups. “Which creative did you like the best, and Why?” is a good open-ended starting point to elicit the information you might be looking for.
How big should a focus group be?
Traditional focus groups need to be small, around 6-8 people. You want to make sure you have enough people to get a diverse set of opinions, but few enough people so that you can hear everyone’s thoughts on each question in a short timeframe.
You’ve run the process with groups of 100+. How do you manage the process with large groups? What are the advantages and challenges of a large group?
Large-scale focus groups are relatively new and very exciting. In a traditional focus group, everyone is usually participating in-person and conversations are one voice at a time, so it takes a while to get everyone’s opinions. In large-scale focus groups, everyone is participating online. So you can get 100+ opinions in the time it would normally take you to get just one. That can be a big advantage.
In a small focus group, you may see a very opinionated, loud person drive everyone else’s responses, or a quiet person having no voice. Online though, everyone is answering the question without hearing anyone else, so you get a diverse set of responses with no outside influence. The challenge is that it is harder to follow up individually. Individual follow-ups can be done to a limited extent, but you generally can’t go as deep as you would be able to in a traditional focus group.
An online focus group is also a lot more scripted. With a good script, this is okay. But if you’re still probing for the “right” questions to ask, then having the ability in a traditional focus group to redirect the moderator may be valuable.
What pitfalls are there that need to be avoided in focus group work?
The biggest pitfall is believing everything you hear. Focus groups – of any size – are directional, and any conclusions should be tested. It’s sometimes tempting to treat what you hear as truth when it’s really just opinion, and it may not be what drives results. You are looking for a match between what you hear in a focus group and the results you see in A/B testing. That is when you know you are going in the right direction.
Can you give any guidance on typical timescales and costs for this type of research?
Focus groups can be relatively inexpensive to run. The largest cost may even be the compensation for the participants, depending on who you’re inviting and what you need to give in order to get them to attend. If you have research partners already picked out and you know who your audience is and what you hope to learn, you can field a focus group in as little as 3 weeks. It’s a tight schedule – but it can be done. The cost is generally in the low tens of thousands of dollars (or pounds).
Focus groups sound like something for big brands only? What can small brands with smaller budgets do?
Focus groups aren’t prohibitively expensive, so small brands shouldn’t rule them out before examining the costs and potential benefit. But if you can’t get the budget approvals, there are other ways to connect with your customers. Doing customer breakfasts can be just as enlightening if you prepare for them in advance. Breakfasts can be very cost efficient if you have someone in the office call local customers and use a local eatery to bring in some bagels and coffee. You’ll still want to budget for a small gift (e.g. a gift card) to the customers who are courteous enough to participate. And of course, these are very directional. Think of this as an opportunity to get some ideas to guide future tests.
What other types of qualitative and quantitative evaluation do you recommend?
My favorite research is pre- and post- brand research. If you want to get a read on an entire program and how it has chanced perceptions, likelihood to purchase, or any number of other factors, this is smart research to plan and budget for. It does take preparation in advance, but you learn so much.
Years ago, my team did this type of research with a client and proved that the value of an unopened email in the inbox can be significant. Because we were able to combine this type of research with actual metrics to quantify the value of an unopened email for our client, we got away from some of the open-rate conversations and instead talked about conversions and attribution.
Andrew Kordek from Trendline Interactive echoed Gretchen’s experience and told me about his experience of focus groups
“Case in point: We had a client years ago that was convinced their content was the bees knees when it came to helping the subscriber journey thru their consideration set in making a large life altering purchase. When we had focus groups of subscribers who were customers and non-customers we found out that what they were truly looking for, in emails from the company, was the exact opposite of what they had been sending for years”
My thanks again to Gretchen and Andrew for being so willing to share their experience.