Three Tips for A/B Testing Success

Last week, Marka and the FEI tribe learned how fire businesses can use A/B testing to improve their direct marketing efforts. This week, the tribe discusses best practices for A/B testing. Remember, fire = print.

Zoot, Marka, Org and Numo were talking about A/B testing over a lovely outdoor lunch at Demeter’s Café.

“A/B testing is basically a series of intelligent marketing experiments,” Zoot said.

“But like almost any experiment, an A/B test should be done according to a certain procedure,” Marka added. “The most successful A/B tests will adhere to these three guidelines, at least: [With that, she took out a stylus—she never left the office without one—and started scribbling on her placemat for the tribe to see.]

1. Only Tests One Variable

“For example,” Marka continued, “last week, we discussed changing up our promotional O-mailer to offer those purchasing our torches a pack of Lucy’s new matches instead of a pair of tickets to the Olympus Fire Festival. In this case, we must ensure that the offer is absolutely the only variable changed in the ‘B’ test. If our ‘B’ test has more than one variable, we won’t be able to determine which variable caused the change in results, if there is a change.”

“The purpose of the A/B test is to identify which tested variable works best,” Org added. “To accurately do so, we have to test one at a time.”

“Exactly,” Marka said. [She returns to writing on her placemat.]

2. Focuses on a Random Sample

“Like all experiment results, the results of A/B tests will be much more useful if the ‘control’ and ‘variable’ tests reach similar, random samples of FEI prospects,” Marka noted.

“If we send the ‘A’ test to a group of relatively unknown prospects, and the ‘B’ to a group of legacy customers, for instance, the results will be skewed, because people from group ‘B’ are probably more likely to respond,” Zoot said.

T.J. is team leader of Grow Sales, Inc., a marketing and social media services company operating at the intersection of compelling content, clear vision and quality communication practices. In this blog, fire is a metaphor for print. Hang on, this ride will be weird...Prometheus crept into Mt. Olympus, stole fire, returned to the lowlands, ran from house to house distributing it, got caught, was chained to a rock, lost his liver to a huge ugly bird and was rescued by Hercules. Leveraging his fame, Prometheus started Fire Enterprises Inc.  (FEI). Since fire was the hottest technology of the time, company success came fast and furious. Two generations later, fire isn't such an easy sale. Now led by Prometheus' grandson Org, FEI's growth is non-existent, competitors are pounding and prices are in the toilet.
Related Content