Control Group Problems for Online Conversion A/B and Bayesian Testing
Control groups are tricky.
And, in related news, new research from Orbitz shows that telepathy is not an effective marketing tool.
As a digital marketing firm, we are often tasked with improving clients’ online conversion rates — getting more leads, adding more subscribers to the company newsletter, increasing views of lead-generation content marketing assets, etc.
This type of effort often involves traditional A/B testing or Bayesian A/B testing, but there is always a control group.
I recently experienced a control group problem that may be more common than many people realize.
Cutting to the chase, the punchline is that I believe I was added to a control group by a large online travel company many years ago, which is all well and good except for the fact that they never removed me from the control group.
Recently, I was talking to a friend who worked at Orbitz. After a while, I said “You know what? I’ll be honest. I very rarely use Orbitz. Expedia and other travel companies are constantly emailing me with travel offers, discount codes and promotions. I know I’ve used Orbitz before but I don’t think they ever follow up with me to make sure I keep doing business with them.”
“Oh,” my friend said. “You probably were added to a control group. Maybe they were testing frequency of email outreach to see what frequencies converted better. It’s possible they put you into some control group for testing and never took you out of it. But believe me, we send out a lot of emails just like Expedia, Travelocity, Priceline and everybody else. How many emails do you get from us anyway?”
Here’s how many emails I’ve gotten from Orbitz in the last two years:
So, even though I’m their customer, I only received 10 emails in 2 years, and only one of those was a promotional email.
How does that compare to, say, Expedia? Here’s a snapshot of what I get from Expedia (side note: they sure do love putting symbols in their email subject lines, don’t they?):
Expedia emails me about five times a week. Based on simple math, they get their brand and their offerings in front of me about 50 times more often than Orbitz does.
Not surprisingly, since they market to me, and Orbitz doesn’t, I use Expedia way more than Orbitz.
As I told my friend, “If I’m in a control group, it’s an odd control group. It’s effectively checking to see how often somebody engages with you if you never, ever market to them.”
More likely, we decided, this was some kind of database error in which I was added to a control group and never removed from it.
The key takeaway for those of you involved with email marketing and conversion rate optimization is that when you set up a control group for A/B testing email subjects, offers, landing pages and the like, don’t forget to take people out of the control group after the experiment is finished.
Interestingly, Orbitz and Expedia are in the process of merging.
I really hope that the merged entity doesn’t keep me in a “let’s market to them via telepathy and see if that works” control group. Personally, I’m fine with receiving five emails a week promoting travel deals.