Three Tips for A/B Testing Success

Last week, Marka and the FEI tribe learned how fire businesses can use A/B testing to improve their direct marketing efforts. This week, the tribe discusses best practices for A/B testing. Remember, fire = print.

Zoot, Marka, Org and Numo were talking about A/B testing over a lovely outdoor lunch at Demeter’s Café.

“A/B testing is basically a series of intelligent marketing experiments,” Zoot said.

“But like almost any experiment, an A/B test should be done according to a certain procedure,” Marka added. “The most successful A/B tests will adhere to these three guidelines, at least: [With that, she took out a stylus—she never left the office without one—and started scribbling on her placemat for the tribe to see.]

1. Only Tests One Variable

“For example,” Marka continued, “last week, we discussed changing up our promotional O-mailer to offer those purchasing our torches a pack of Lucy’s new matches instead of a pair of tickets to the Olympus Fire Festival. In this case, we must ensure that the offer is absolutely the only variable changed in the ‘B’ test. If our ‘B’ test has more than one variable, we won’t be able to determine which variable caused the change in results, if there is a change.”

“The purpose of the A/B test is to identify which tested variable works best,” Org added. “To accurately do so, we have to test one at a time.”

“Exactly,” Marka said. [She returns to writing on her placemat.]

2. Focuses on a Random Sample

“Like all experiment results, the results of A/B tests will be much more useful if the ‘control’ and ‘variable’ tests reach similar, random samples of FEI prospects,” Marka noted.

“If we send the ‘A’ test to a group of relatively unknown prospects, and the ‘B’ to a group of legacy customers, for instance, the results will be skewed, because people from group ‘B’ are probably more likely to respond,” Zoot said.

“Right,” Marka agreed. [Again turning to the placemat.]

3. Tests a Specific “Hypothesis”

“Good A/B tests should contain a ‘B’ element that’s well thought out and will help illuminate a question we have about our current direct marketing activities,” Marka explained. “In this case, the question we seek to answer is, ‘Will the offer of a new, high-value, but low-cost (to us), FEI product generate more recipient interest than a relatively more pricey non-FEI incentive?’ Our working hypothesis, then, is that offering matches will lead to not only better response rates but, eventually, better brand awareness and sales for this item.”

“Sounds like we’re doing everything right,” Org observed, shoveling a forkful of Greek Salad into his mouth.

“I’d say so,” Marka said. “But Org...” [She looked at the FEI leader and rubbed her face..] “You might want to get out a napkin.”

Today’s FIRE! Point
Like almost any experiment, an A/B test has to be done according to a certain procedure. The most successful A/B tests will only test one variable, be focused on a random sample, and contain an intelligent hypothesis.

Campaign Monitor Improves Click-through Results with A/B Testing
The e-mail marketing software company sent two versions of an e-mail with a link to a survey that they wanted recipients to complete. CM changed only one line—the call-to-action in the “B” e-mail. The results: their original version “A” actually received 51 percent more click-throughs.

Next week: The FEI tribe shifts gears to product strategy and discusses when to enter a new product category vs. extend an existing product line.