Three Tips for A/B Testing Success

Last week, Marka and the FEI tribe learned how fire businesses can use A/B testing to improve their direct marketing efforts. This week, the tribe discusses best practices for A/B testing. Remember, fire = print.

Zoot, Marka, Org and Numo were talking about A/B testing over a lovely outdoor lunch at Demeter’s Café.

“A/B testing is basically a series of intelligent marketing experiments,” Zoot said.

“But like almost any experiment, an A/B test should be done according to a certain procedure,” Marka added. “The most successful A/B tests will adhere to these three guidelines, at least: [With that, she took out a stylus—she never left the office without one—and started scribbling on her placemat for the tribe to see.]

1. Only Tests One Variable

“For example,” Marka continued, “last week, we discussed changing up our promotional O-mailer to offer those purchasing our torches a pack of Lucy’s new matches instead of a pair of tickets to the Olympus Fire Festival. In this case, we must ensure that the offer is absolutely the only variable changed in the ‘B’ test. If our ‘B’ test has more than one variable, we won’t be able to determine which variable caused the change in results, if there is a change.”

“The purpose of the A/B test is to identify which tested variable works best,” Org added. “To accurately do so, we have to test one at a time.”

“Exactly,” Marka said. [She returns to writing on her placemat.]

2. Focuses on a Random Sample

“Like all experiment results, the results of A/B tests will be much more useful if the ‘control’ and ‘variable’ tests reach similar, random samples of FEI prospects,” Marka noted.

“If we send the ‘A’ test to a group of relatively unknown prospects, and the ‘B’ to a group of legacy customers, for instance, the results will be skewed, because people from group ‘B’ are probably more likely to respond,” Zoot said.

T.J. Tedesco is a sales growth, business strategy, marketing and PR consultant operating at the intersection of clear vision, compelling content and effective outreach practices. For nearly two decades, T.J. has been an independent consultant and sales growth team leader. Previously, he sold commercial printing, graphic arts machinery and supplies, and finishing and bindery services. T.J. helps North American companies with content development, Web and print design leadership, nurture marketing programs, sales coaching, sales team alignment and business strategy. Since 1996, T.J. has worked with more than 100 clients on retainer, 80 percent in the graphic arts industry. T.J. is author of "Win Top-of-Mind Positioning," "Playbook for Selling Success in the Graphic Arts Industry," "Fire! How Marketing Got Hot," "Direct Mail Pal" and four more books published by PIA. He can be reached at (301) 404-2244 or
Related Content