Three Tips for A/B Testing Success

Last week, Marka and the FEI tribe learned how fire businesses can use A/B testing to improve their direct marketing efforts. This week, the tribe discusses best practices for A/B testing. Remember, fire = print.

Zoot, Marka, Org and Numo were talking about A/B testing over a lovely outdoor lunch at Demeter’s Café.

“A/B testing is basically a series of intelligent marketing experiments,” Zoot said.

“But like almost any experiment, an A/B test should be done according to a certain procedure,” Marka added. “The most successful A/B tests will adhere to these three guidelines, at least: [With that, she took out a stylus—she never left the office without one—and started scribbling on her placemat for the tribe to see.]

1. Only Tests One Variable

“For example,” Marka continued, “last week, we discussed changing up our promotional O-mailer to offer those purchasing our torches a pack of Lucy’s new matches instead of a pair of tickets to the Olympus Fire Festival. In this case, we must ensure that the offer is absolutely the only variable changed in the ‘B’ test. If our ‘B’ test has more than one variable, we won’t be able to determine which variable caused the change in results, if there is a change.”

“The purpose of the A/B test is to identify which tested variable works best,” Org added. “To accurately do so, we have to test one at a time.”

“Exactly,” Marka said. [She returns to writing on her placemat.]

2. Focuses on a Random Sample

“Like all experiment results, the results of A/B tests will be much more useful if the ‘control’ and ‘variable’ tests reach similar, random samples of FEI prospects,” Marka noted.

“If we send the ‘A’ test to a group of relatively unknown prospects, and the ‘B’ to a group of legacy customers, for instance, the results will be skewed, because people from group ‘B’ are probably more likely to respond,” Zoot said.