Control Group Problems for Online Conversion A/B and Bayesian Testing

Marketing with Mental Telepathy

Control groups are tricky.

And, in related news, new research from Orbitz shows that telepathy is not an effective marketing tool.

As a digital marketing firm, we are often tasked with improving clients’ online conversion rates — getting more leads, adding more subscribers to the company newsletter, increasing views of lead-generation content marketing assets, etc.

This type of effort often involves traditional A/B testing or Bayesian A/B testing, but there is always a control group.

I recently experienced a control group problem that may be more common than many people realize.

Cutting to the chase, the punchline is that I believe I was added to a control group by a large online travel company many years ago, which is all well and good except for the fact that they never removed me from the control group.

Recently, I was talking to a friend who worked at Orbitz. After a while, I said “You know what? I’ll be honest. I very rarely use Orbitz. Expedia and other travel companies are constantly emailing me with travel offers, discount codes and promotions. I know I’ve used Orbitz before but I don’t think they ever follow up with me to make sure I keep doing business with them.”

“Oh,” my friend said. “You probably were added to a control group. Maybe they were testing frequency of email outreach to see what frequencies converted better. It’s possible they put you into some control group for testing and never took you out of it. But believe me, we send out a lot of emails just like Expedia, Travelocity, Priceline and everybody else. How many emails do you get from us anyway?”

Here’s how many emails I’ve gotten from Orbitz in the last two years:

Orbitz Email Marketing

So, even though I’m their customer, I only received 10 emails in 2 years, and only one of those was a promotional email.

How does that compare to, say, Expedia? Here’s a snapshot of what I get from Expedia (side note: they sure do love putting symbols in their email subject lines, don’t they?):

Expedia Email Marketing

Expedia emails me about five times a week. Based on simple math, they get their brand and their offerings in front of me about 50 times more often than Orbitz does.

Not surprisingly, since they market to me, and Orbitz doesn’t, I use Expedia way more than Orbitz.

As I told my friend, “If I’m in a control group, it’s an odd control group. It’s effectively checking to see how often somebody engages with you if you never, ever market to them.”

More likely, we decided, this was some kind of database error in which I was added to a control group and never removed from it.

The key takeaway for those of you involved with email marketing and conversion rate optimization is that when you set up a control group for A/B testing email subjects, offers, landing pages and the like, don’t forget to take people out of the control group after the experiment is finished.

Interestingly, Orbitz and Expedia are in the process of merging.

I really hope that the merged entity doesn’t keep me in a “let’s market to them via telepathy and see if that works” control group. Personally, I’m fine with receiving five emails a week promoting travel deals.

Ken Gaebler founded Walker Sands as a full-service marketing firm back in 2002 with the goal of creating an exceptional more

Write a Reply or Comment

Your email address will not be published. Required fields are marked *

Newsletter Sign Up

Please leave this field empty.

Recent Posts

One Way Small Companies Beat Big Companies: By Telling It Like It Is

On a recent trip to Silicon Valley, while waiting in the lobby of Digital Garage to meet an old frie...

Pitching a Big Idea

Here's everything you need to know about pitching a big idea, including how to overcome 32 common ob...

You’re Underinvesting in Your Personal Thought Leadership

You work for a brand that everyone knows and respects. But will it be around forever? Will you be wi...

Does Your Demand Gen Program Make People Unhappy? It Should!

If you work in demand generation, making people unhappy is your job. Admittedly, it's not fun to ...

Reflections on a Walker Sands Training Session

Earlier today, I attended one of our Walker Sands offsite training sessions. This was an event fo...