Most direct marketers find statistical formulas difficult to understand. Many respond by ignoring them. Or, they take the easy road and heed the advice of consultants who say things such as, "You're fine as long as you have 50 responders."
This widespread lack of sophistication costs direct marketers dearly in squandered opportunity, in the form of unrealized revenue. In this article, we'll illustrate the magnitude of the problem via a two part-statistical formula. This formula, which was first presented in "How Big Should My Test Be?" (DM News, October 1, 2001), assists in determining the quantity of names to include in test panels. We'll see how inputting this formula into a spreadsheet, and investing an hour or two in experimentation, might just generate millions of dollars for your direct marketing business.
A Quick Review of the October Article
The following is the formula for determining how many names to include in a test panel:
Part 1: (Expected Response Rate X (1 - Expected Response Rate) X Z squared) / Precision squared
Part 2: Answer to Part 1 / (1 + (Answer to Part 1 / Rollout Universe Quantity))
Understanding "Z" would require a statistics lesson. All we need to know for our purposes, however, is that it corresponds to the level of confidence that we have in the accuracy of our test panel response rate. For example, a given test panel quantity might result in a confidence level of 90% that a test panel response rate of 0.8% will translate to a rollout rate of between 0.72% and 0.88%.
The following are six combinations of Z and confidence levels: a Z of 1.96 corresponds to a confidence level of 95%, a Z of 1.645 to 90%, 1.282 to 80%, 1.04 to 70%, 0.84 to 60%, and 0.67 to 50%.
Precision describes the degree of "plus/minus" uncertainty around a test panel response rate. We can never know for sure from a test panel response rate what the "true" rollout rate will be. For example, with a test panel response rate of 0.8% and a universe size of 100,000, a test panel size of 5,273 will result in our being 50% confident that the rollout response rate will be between 0.72% and 0.88%. In other words, one out of every two times the rollout rate will be within ten percent of the test panel rate.
You can check this yourself by substituting "0.67" for "Z" in the formula, and "0.08%" for "Precision." You'll know that you've done it correctly when you get an answer of "5,273."
We've just established that one half of the time the true rollout response rate will be between 0.72% and 0.88%. By definition, one out of every two times it will be outside of this range. Therefore, by extension, one out of every four times the rollout rate will be less than 0.72%.
Generating Millions of Dollars
For most direct marketers, the effort involved in understanding this formula is less pleasant than tasks such as sourcing new merchandise and working on promotional layouts. However, it's just as important. We'll build upon the example from the previous section to illustrate why this formula might just generate millions of dollars for your direct marketing business.
In the example, we established that one out of every four times the true rollout response rate will be under 0.72%, one out of two times it will be between 0.72% and 0.88%, and one out of four times it will be over 0.88%. Of course, in the real world there is no way to know for sure what the true rollout response rate is without going through the effort of contacting everyone.
However, let's assume for the sake of illustration that we magically know in advance that the true rollout response rate is identical to the test panel rate of 0.8%. Although extremely rare in direct marketing testing, it does happen on occasion that the two are the same. Let's also assume that 0.72% "or, ten percent less than 0.8% "is the minimum test panel response rate that is required for rollout.
Armed with this information, we know that one out of every four times a test panel size of 5,273 will result in our failing to roll out the list select, because the test panel response rate will be below the required 0.72%. This is a missed opportunity because the true rollout response rate of 0.8% is comfortably above our minimum. The financial ramifications of this missed opportunity are profound because rollouts generally are repeated many times.
Let's assume that we contact proven rental lists three times a year. By failing to roll out the 100,000 list select, we will have failed to cost-effectively generate 300,000 promotions a year, or 1.5 million over a five-year period. Using our response rate assumption, that's 12,000 missed customers!
Now, let's assume that each new customer will, on average, order one additional time, and that the size of each order is $90. That translates into 24,000 missed orders and $2.16 million of missed revenue over the five years. And, that's from just a single missed rollout!
There's no right or wrong answer when deciding on test panel quantities. In other words, there's no one size that will be optimal for every direct marketer. The appropriate quantity will depend on factors such as the amount of money available for testing, and the level of risk the direct marketer is willing to assume that the rollout response rate will be significantly different from the test rate. Nevertheless, most direct marketers are appalled when they are made to understand the profound financial ramifications of small test panel quantities.
Fortunately in the example earlier, it's possible to minimize the risk of failing to identify the $2.16 million in revenue. What has to be done is to increase the size of the test panel. Let's explore the effects of various panel quantities on the accuracy of our test reads. Throughout, we'll assume that promotional costs, including list rental, are $1.00 per thousand:
If we increase our test panel quantity from 5,273 to 8,046, we'll spend an extra $2,763. With a response rate of 0.8%, we'll increase our responder quantity from 42 to 64. As a result, we'll fail to identify the $2.16 million dollar opportunity just one out of every five times rather than one out of every four.
If we move from 5,273 to 11,826, we'll invest an extra $6,543 in order to fail to identify the $2.16 million just fifteen percent of the time. That's going from 42 to 95 responders. Likewise, with 16,930, we'll spend an incremental $11,647 to fail just one out of ten times. Finally, with 25,124, we'll invest an extra $19,841 to fail just one out of twenty times.
Final Thoughts
You'll have to decide for yourself which of these quantities and corresponding costs is the best balance for you. As the test quantities increase, the chances of failing to identify the $2.16 million decrease. At the same time, the costs of testing increase. There is a specific point that reflects your personal equilibrium.
Be mindful throughout that the incremental costs presented earlier for increased certainty are based on gross rather than net calculations. Orders will be generated from the additional promotional quantities, which will defray a portion of the costs. And, when you are more confident in your results, you can proceed to full rollout more quickly.
Most direct marketers have no idea of the extent of the missed opportunities that result from small test panels. This is understandable because missed opportunities are "by definition "a hidden phenomenon. Out of sight, out of mind, so to speak. Nevertheless, direct marketers who are unwilling to understand the basics of statistical sampling theory are paying a steep tax for their lack of knowledge.
Unfortunately, the list brokerage industry "those professionals whose job it is to make test recommendations to the direct marketing community "generally do little to educate their clients on these issues. This is because most list professionals are no more comfortable with sampling theory than their clients. However, that's the subject for another article!
