Test-and-learn analytics does not contradict or compete with traditional data analytics, it builds thereupon. All test-and-learn programmes start with a traditional analysis of historical data. However, rather than just using this information to make an informed hypothesis about the future, it constructs an environment in which that hypothesis can be scientifically verified before being widely accepted or rejected.

The first benefit of this approach is that it creates a true measure of the performance differential between two or more strategies. In an environment where only one strategy at a time exists, it is impossible to know to what degree any observed difference in performance is due to the strategy (internal and controllable) and to what degree it is due to the environment (external and uncontrollable).

Consider an internet florist who sells roses at a 10% discount in the month leading up to Valentine’s Day. If he experiences an increase in sales, how much of that is due to the lower price and how much is due to the general increase in demand for roses at that time of year? Because he only has one strategy each year it is impossible to determine. This would be true even if he offered a 10% discount in one year and a 15% discount in the next. It would once again be impossible to correctly attribute any increase in sales in the second year between the change in the discount and any changes in the larger economy. If however, in the first week of the campaign, he were to randomly offer half of the visitors to his website a 10% discount and the other half a 15% – and this is easy to do on the internet – he could calculate the degree to which an increase in discount relates to an increase in sales. This is test-and-learn in action.

The second major benefit of the test-and-learn approach is that it controls the costs and risks that arise whenever a new strategy is implemented. Let’s assume that our florist only experiences a very small increase in sales when the larger discount is offered. If he had offered the 15% discount for the whole month he would have lost money because the larger discount would not have been sufficiently compensated for by increased sales. However, by testing the two offers for a week he has only ‘lost’ money on half of a week’s sales and is able to run the more profitable strategy for the remaining three weeks of the campaign. Innovation has therefore become cheaper and safer so more ideas can be tested which increases the odds of finding a new competitor-beating strategy. It is possible to see how our florist, because he can now measure new strategies without risking his whole month’s sales, could try to optimise sales by changing the wording of his advertisements, by using new photos, etc.

Thirdly, when the test-and-learn approach is fully embedded within and organisation’s DNA it naturally leads to continuous improvement. Test-and-learn analytics is a circular process without end. An idea, or hypothesis, is first tested against the evidence from a scientific experiment and then either implemented or rejected. But, instead of stopping there, the process regularly repeats itself. As soon as one idea has been implemented it must be tested against the next idea and then the next idea after that. At the end of each cycle the strategy is either improved or an inferior alternative is cheaply discarded – both of which strengthen the organisation as a whole.

But unless test-and-learn is fully understood at all levels of an organisation it can lead to dangerous misinformation. Therefore, the mechanics of every test must be understood before actions are taken based upon the results thereof. The three most important questions to ask of any piece of analysis are: Is the business model fully understood? Are the test and control groups statistically identical? Are the results proven or just implied?

Most organisations consist of several interconnected profit levers – some of which compete with one another and others of which enable one another. The profit of such an organisation is maximised when all their profit levers act together in the optimal way. Unless the test has considered the implication of a proposed strategy on all the relevant profit levers it might improve one part of the business at the cost of another. For example, a simpler loan application process may increase response rates but might also increase risk and so both of these measures need to be included in the test.

Statistically identical test and control groups are created by randomly assigning candidates to one or the other. If this assignment is not done correctly, certain underlying trends can unknowingly be built into the test. For example, although you can use the last digit of a credit card number to randomly assign groups, using the whole number would group similar candidates together – those who bank together, who have similar incomes, etc.

Much has been made in this article about the fact that test-and-learn analytics leads to scientifically proven results. This is, however, only true for the aspects specifically measured in the test. Broader implications that may have been drawn from the test results are only as accurate as the assumptions through which they were derived. For example, a test might prove that one month after selling a credit card to a new group of customers, those customers are no riskier than traditional customers. And, it might then be assumed that this new group of customers will still be equally risky in a year’s time. Although this assumption appears reasonable, it has not been specifically proven one and so, in time, it might be shown to be inaccurate. Long-term strategies based on short-term tests must therefore be undertaken with caution.

on September 3, 2010 at 9:27 AM |Profit Model Analytics – A Banking Example « Credit Risk Strategy[…] performance’ and can only be calculated using a forward-looking analytical technique. Test-and-learn analytics is a technique that gathers real-time marginal performance data in a series of small and […]