Control Group Problems for Online Conversion A/B and Bayesian Testing

Marketing with Mental Telepathy

Control groups are tricky.

And, in related news, new research from Orbitz shows that telepathy is not an effective marketing tool.

As a digital marketing firm, we are often tasked with improving clients’ online conversion rates — getting more leads, adding more subscribers to the company newsletter, increasing views of lead-generation content marketing assets, etc.

This type of effort often involves traditional A/B testing or Bayesian A/B testing, but there is always a control group.

I recently experienced a control group problem that may be more common than many people realize.

Cutting to the chase, the punchline is that I believe I was added to a control group by a large online travel company many years ago, which is all well and good except for the fact that they never removed me from the control group.

Recently, I was talking to a friend who worked at Orbitz. After a while, I said “You know what? I’ll be honest. I very rarely use Orbitz. Expedia and other travel companies are constantly emailing me with travel offers, discount codes and promotions. I know I’ve used Orbitz before but I don’t think they ever follow up with me to make sure I keep doing business with them.”

“Oh,” my friend said. “You probably were added to a control group. Maybe they were testing frequency of email outreach to see what frequencies converted better. It’s possible they put you into some control group for testing and never took you out of it. But believe me, we send out a lot of emails just like Expedia, Travelocity, Priceline and everybody else. How many emails do you get from us anyway?”

Here’s how many emails I’ve gotten from Orbitz in the last two years:

Orbitz Email Marketing

So, even though I’m their customer, I only received 10 emails in 2 years, and only one of those was a promotional email.

How does that compare to, say, Expedia? Here’s a snapshot of what I get from Expedia (side note: they sure do love putting symbols in their email subject lines, don’t they?):

Expedia Email Marketing

Expedia emails me about five times a week. Based on simple math, they get their brand and their offerings in front of me about 50 times more often than Orbitz does.

Not surprisingly, since they market to me, and Orbitz doesn’t, I use Expedia way more than Orbitz.

As I told my friend, “If I’m in a control group, it’s an odd control group. It’s effectively checking to see how often somebody engages with you if you never, ever market to them.”

More likely, we decided, this was some kind of database error in which I was added to a control group and never removed from it.

The key takeaway for those of you involved with email marketing and conversion rate optimization is that when you set up a control group for A/B testing email subjects, offers, landing pages and the like, don’t forget to take people out of the control group after the experiment is finished.

Interestingly, Orbitz and Expedia are in the process of merging.

I really hope that the merged entity doesn’t keep me in a “let’s market to them via telepathy and see if that works” control group. Personally, I’m fine with receiving five emails a week promoting travel deals.

Ken Gaebler founded Walker Sands as a full-service marketing firm back in 2002 with the goal of creating an exceptional more

Write a Reply or Comment

Your email address will not be published. Required fields are marked *

Newsletter Sign Up

Please leave this field empty.

Recent Posts

The Power of Automation: How Different SEM Tools Ensure Your Ads Stay Relevant

Ad relevance is a crucial factor in how well your search campaigns perform.  Focusing on ad copy re...

Charles Adler, Co-Founder of Kickstarter, Is Opening a Creative Space in Chicago and This Is What it Looks Like

On Friday I finally got a chance to check out a Creative Mornings event. These things -- I think the...

A Primer on Predictive Marketing for the Chief Sales Officer

Here's a marketing prediction I can make with a high level of certainty. True of false -- as a Ch...

Salvaging Content from an Old Blog

This is an article about what to do with that old, cobweb-filled corporate blog of yours. You kno...

SEO Tips for CEOs and Marketers: Improve Your Digital Marketing Strategy by Changing How You Measure Conversions Pt. 2

Last time, we discussed how to think about measuring conversions and the ins and outs of Google Anal...