Archive for the ‘Test-and-Learn’ Category

An effective knowledge management strategy is mandatory for any organisation wanting to succeed in today’s knowledge-based economy.  Such strategies should cover the creation of knowledge as well as the systems that allow for it to be stored, shared and used as a catalyst for the creation of further knowledge.


That said, regardless of the form that shared knowledge portals take, they remain stubbornly under-used and under-stocked.  This is likely to remain the case for as long as they require staff to shift their effort away from their immediate activities to find, read and interpret other peoples’ work.  A knowledge management strategy built around tools that sit outside of the day-to-day activities of an organisation’s staff is unlikely to add real value


But, by developing a culture of test-and-learn analytics, it is possible to entrench knowledge within its “organisational DNA”.  In so doing, that knowledge becomes easier to store, easier to share and easier to access.



Test-and-learn is a simple but often misapplied concept.  When it ingrained within the culture of an organisation, however, it can deliver excellent financial and knowledge management results.  Based on the scientific method, it first came to prominence as a business concept in the late eighties when American credit card issuers used it to dramatically grow their industry.


Test-and-learn analysis is an evidence-based technique whose starting point is always the hypothesis that a proposed new strategy will be more profitable than the incumbent strategy.  This expectation is usually based on an analysis of existing data stored in in-house databases but could also include experience “stored” within the minds of staff members and information acquired from third-parties.


But business decisions should not be made on hypotheses alone and so theses hypotheses must first be tested.  It is from the results of this testing that learnings are gained.  The test must compare the proposed strategy to the incumbent one in a controlled environment free from extraneous influences.  The results of the test are monitored are then subjected to statistical analysis to identify the more profitable of the two strategies.  Thus identified, the ‘winning strategy’ is rolled-out across the board and becomes the incumbent strategy against which any future hypotheses are to be tested.


Consider a marketing analyst working for a retail bank who must chose between two potential marketing campaigns designed to generate applications for credit cards.  Traditionally, the bank’s new customers were enticed with the offer of a year’s free membership to its loyalty programme.

Our analyst, however, hypothesises that more customers would apply for a card if the bank offered to waive the card fees for the first year.  Wanting to make the best use of her limited budget, she must first design a test to prove her hypothesis.  A portion of potential customers will each be randomly offered one of the two options.  After two months of careful analysis, she will be able to prove whether customers respond better to her “no fees” offer.  This real evidence will justify her using the bulk of her marketing budget to advertise her “no fees” offer.  The successful strategy then also becomes the standard against which all future marketing strategies are to be compared.


The test-and-learn approach therefore creates a circular pattern as it moves an unproven hypothesis from theory to established fact against which, in time, new hypothesis will be tested.  This circular nature is what makes the test-and-learn approach a good knowledge management tool.


Creating Explicit and Collective Knowledge

The example began with tacit individual knowledge in the form of a marketing analyst’s hypothesis that a “no fees” offer would improve response rates.  By testing this hypothesis in a scientific manner, she was able to turn that tacit knowledge into explicit knowledge.  At this stage though, that explicit knowledge was only held by the analyst running the test – i.e. individual explicit knowledge.  However, as soon as the learnings from the test were used to change the marketing strategy, everyone who came into contact with the new strategy would also become explicitly aware of the new knowledge.


The test-and-learn process can therefore be simplified into three stages – “developing the hypothesis”, “testing the hypothesis” and “taking action”.  Each of these stages, in turn, represents a stage in the knowledge management process – “tacit individual knowledge” becomes “explicit individual knowledge” and then “explicit collective knowledge”. 


So, once the culture of test-and-learn analytics is fully embedded within an organisation, any member of staff need only look at test outcomes to have de facto access to all the relevant knowledge of that organisation.  Because the “free loyalty programme” campaign was replaced by the “no fees” campaign anyone with an interest in the organisation’s marketing strategies will know that the “no fees” offer is better.  Every time this knowledge is updated, for example if it is subsequently found that response rates do not drop if it is only the first six months’ worth of fees that are waived, those same stakeholders will once again have access to the new knowledge when the strategy is changed again.

Read Full Post »

Test-and-learn analytics is a conceptually simple technique that is often misunderstood in practice.  In fact, many organisations fail in their test-and-learn endeavours before they even start – by incorrectly defining the concept. 


Simply rolling out a new strategy to all customers and monitoring what happens is not “test-and-learn”.  That’s trial-and-error and it is an expensive and reckless business practice that provides little detailed information and no mitigation for the risk of failure. 


Test-and-learn analytics, on the other hand, limits the costs and risks of innovation by testing all new strategies against incumbent strategies in a controlled environment before committing to a large-scale roll-out.  When properly implemented, test-and-learn analytics can change the way that an organisation – and even whole industry – does business and it can deliver outstanding results.  Such a successful implementation is built on three pillars: the right culture; the right people and the right tools.



Once the leadership of an organisation has decided to the adopt test-and-learn analytics, the most common mistake they make is to see it as a technical issue to be dealt with by the IT department.  While it is true that test-and-learn analytics require certain technical enablers – which will be discussed later – the single most important requirement for ensuring a successful implementation is the right organisational culture.


The successful adoption of test-and-learn requires the organisation’s culture to apply two distinct forces.  Firstly, senior business decision-makers must demand the results of test-and-learn analysis as an input into all their important decisions.  When alternative strategies are being raised at the board level, the directors must demand comparative test results before giving their go ahead.  When budgetary approval is being sought for a major initiative, the finance team must demand the results of pilot test before signing-off.  Secondly, test-and-learn thrives in a flat structure where ideas are evaluated on their own merits, independent of the relative seniority of the person putting them forward.  Business expertise should of course lead to more astute assumptions at the start of the process but, once the process is complete and the results of the test have been analysed, these test results should be allowed to speak for themselves.This allows alternative proposals to compete on a level playing field which, in turn, means that the decisions that are made are more likely to be the correct ones. 


This move towards a meritocracy is not always a move that is easily made within large organisations but, unless senior management make a concerted effort to flatten their structures, the results of tests will become meaningless and the benefits of the technique will evaporate.



The next consideration for a test-and-learn implementation must be the people.  While traditional analysts could rely on technical know-how; test-and-learn analysts need a broader combination of technical and business skills – as, indeed, do all other stakeholders in the process.  In other words, it is no longer sufficient to just answer the questions posed, analysts need to help their organisations ask and answer the right questions.


It is therefore important to recruit staff with this mix of skills – ideally by taking candidates through a series of numerical case studies dealing with realistic business problems.  These case studies should test a candidate’s ability to identify the profit levers within a business model, construct the equations that describe the interaction of those levers and manipulate them to obtain the relevant information. This ability to translate business problems into solvable equations is far more important to a test-and-learn analyst than a deep understanding of statistical tools as the latter can be taught far more easily.



The market place offers a range of specialised systems that can enable and accelerate large test-and-learn roll-outs but they are all based on the same simple system requirements.


Good quality data is the foundation on which test-and-learn analytics is built and so, in order to provide this, the most important component of the system is an accurate and easily accessed database.  The database should be large enough to store all relevant data for as long as it is needed and designed in a way that facilitates quick and reliable data retrieval.  Databases only hold historical data but to create, implement and measure a test that data needs to first be manipulated.  To achieve this it is important that some form of analytical software sit on top of that database.  Although Microsoft Excel will often suffice in smaller and/ or newer implementations, in time it will likely become necessary to upgrade to the more sophisticated offerings from Experian, SAS, etc.


Once a test has been designed it must be run in real-time as part of the organisation’s day-to-day business operations.  This requires some form of real-time delivery engine.  The nature of an organisations business and the relative level of sophistication it requires will ultimately dictate the level of investment required in such a delivery engine.  However, with careful manipulation of available systems it is often possible to run complex tests using rudimentary systems.


Finally, any insights gained from the data and proven in the test must be communicated to the relevant stakeholders.  Although other products do exist, widely available programmes like Microsoft’s Power Point and Visio are usually sufficient – in the right hands of course – to convey the message in a convenient and effective manner.

Read Full Post »

Test-and-learn analytics does not contradict or compete with traditional data analytics, it builds thereupon.  All test-and-learn programmes start with a traditional analysis of historical data.  However, rather than just using this information to make an informed hypothesis about the future, it constructs an environment in which that hypothesis can be scientifically verified before being widely accepted or rejected.


The first benefit of this approach is that it creates a true measure of the performance differential between two or more strategies.  In an environment where only one strategy at a time exists, it is impossible to know to what degree any observed difference in performance is due to the strategy (internal and controllable) and to what degree it is due to the environment (external and uncontrollable).


Consider an internet florist who sells roses at a 10% discount in the month leading up to Valentine’s Day.  If he experiences an increase in sales, how much of that is due to the lower price and how much is due to the general increase in demand for roses at that time of year?  Because he only has one strategy each year it is impossible to determine.  This would be true even if he offered a 10% discount in one year and a 15% discount in the next.  It would once again be impossible to correctly attribute any increase in sales in the second year between the change in the discount and any changes in the larger economy.  If however, in the first week of the campaign, he were to randomly offer half of the visitors to his website a 10% discount and the other half a 15% – and this is easy to do on the internet – he could calculate the degree to which an increase in discount relates to an increase in sales.  This is test-and-learn in action.


The second major benefit of the test-and-learn approach is that it controls the costs and risks that arise whenever a new strategy is implemented.  Let’s assume that our florist only experiences a very small increase in sales when the larger discount is offered.  If he had offered the 15% discount for the whole month he would have lost money because the larger discount would not have been sufficiently compensated for by increased sales.  However, by testing the two offers for a week he has only ‘lost’ money on half of a week’s sales and is able to run the more profitable strategy for the remaining three weeks of the campaign.  Innovation has therefore become cheaper and safer so more ideas can be tested which increases the odds of finding a new competitor-beating strategy.  It is possible to see how our florist, because he can now measure new strategies without risking his whole month’s sales, could try to optimise sales by changing the wording of his advertisements, by using new photos, etc.


Thirdly, when the test-and-learn approach is fully embedded within and organisation’s DNA it naturally leads to continuous improvement.  Test-and-learn analytics is a circular process without end.  An idea, or hypothesis, is first tested against the evidence from a scientific experiment and then either implemented or rejected.  But, instead of stopping there, the process regularly repeats itself.  As soon as one idea has been implemented it must be tested against the next idea and then the next idea after that.  At the end of each cycle the strategy is either improved or an inferior alternative is cheaply discarded – both of which strengthen the organisation as a whole.


But unless test-and-learn is fully understood at all levels of an organisation it can lead to dangerous misinformation.  Therefore, the mechanics of every test must be understood before actions are taken based upon the results thereof.  The three most important questions to ask of any piece of analysis are: Is the business model fully understood? Are the test and control groups statistically identical? Are the results proven or just implied?


Most organisations consist of several interconnected profit levers – some of which compete with one another and others of which enable one another.  The profit of such an organisation is maximised when all their profit levers act together in the optimal way.  Unless the test has considered the implication of a proposed strategy on all the relevant profit levers it might improve one part of the business at the cost of another.  For example, a simpler loan application process may increase response rates but might also increase risk and so both of these measures need to be included in the test.


Statistically identical test and control groups are created by randomly assigning candidates to one or the other.  If this assignment is not done correctly, certain underlying trends can unknowingly be built into the test.  For example, although you can use the last digit of a credit card number to randomly assign groups, using the whole number would group similar candidates together – those who bank together, who have similar incomes, etc. 


Much has been made in this article about the fact that test-and-learn analytics leads to scientifically proven results.  This is, however, only true for the aspects specifically measured in the test.   Broader implications that may have been drawn from the test results are only as accurate as the assumptions through which they were derived.  For example, a test might prove that one month after selling a credit card to a new group of customers, those customers are no riskier than traditional customers.  And, it might then be assumed that this new group of customers will still be equally risky in a year’s time.  Although this assumption appears reasonable, it has not been specifically proven one and so, in time, it might be shown to be inaccurate.  Long-term strategies based on short-term tests must therefore be undertaken with caution.

Read Full Post »

The test-and-learn approach to strategy optimisation first came to prominence as a business tool twenty years ago in the American credit card industry.  Based on the scientific method, it applies the concept of experimentation to the business environment. 


In short, it compares the results of two or more competing strategies run in a controlled environment free of any extraneous influences.  However, despite its conceptual simplicity, it is often poorly implemented – even by large corporations with advanced technical capabilities. 


A good scientist will carefully plan, implement and measure every aspect of their experiment.  Similarly, for test-and-learn analytics to deliver results reliable enough to be adopted by a business, the analyst must carefully plan, implement and measure every stages of their business experiment.



The planning stage should start with an awareness of the problem or opportunity at hand.  This awareness usually evolves out of an analysis of the organisation’s proprietary data but could also come from information acquired from third-parties.  Only once the business problem has been fully understood can it be articulated in the form of a hypothesis – a statement of expected outcomes.


Any business application of the test-and-learn technique should focus on the optimisation of one or more aspects of that business’s profit model.  Therefore, any hypothesis should be built around the expected interaction of key profit drivers.  For example, a direct marketer may hypothesise that a more colourful envelope will lead to higher sales from a given advertising campaign.  The profit assumption inherent in this hypothesis is that colourful envelopes are more likely to be noticed, opened and read and are thus likely to lead to higher response rates and ultimately to increased sales sufficient to cover the extra cost of those envelopes.


A test must then be designed to conclusively prove or disprove this hypothesis.  The test in this example must be able to prove the degree to which the response rate for a particular advertisement increases when that advertisement is sent in a colourful envelope.


Once the key test measures have been established, the experimenter must select their inclusion and exclusion criteria.  These criteria limit the test to a specific population of interest.  In the example begun above, the experimenter may choose to include only those potential customers with a clean credit record and to exclude any potential customers who have received a previous advertisement from their company within the last three months.



The final step in the planning process is to ensure that the results of the test will be statistically significant.  This is achieved, primarily, by including a sufficiently large population of customers in the test.  Bear in mind however that, since the test-and-learn technique identifies the single most profitable strategy, candidates exposed to the other strategies are value-destroying.  For this reason, the test should included only as many candidates as are necessary to ensure statistical significance.  This also limits the costs and risk of the test.  On the other hand, where there are too few candidates to achieve statistical significance, the experimenter should broaden their inclusion and exclusion criteria.  In our example, this could be done by no longer excluding recently contacted customers.



Implementing a well-planned test should then be a simple process.  The qualifying candidates are randomly assigned to one of two groups – the test group or the control group.  Candidates in the test group will be exposed to the new strategy while those in the control group will be treated as they have always been treated.  Everything else is kept consistent across the groups.  Returning to our example this means that both groups will receive the same advertisement but the test group will receive their’s in a new colourful envelope while the control group will receive their’s in a traditional white one.


Many experimenters underestimate the importance of a true random assignment of candidates between groups.  Assigning candidates to groups randomly ensures that the groups are effectively identical.  Unless the groups are identical, any measured difference in performance might be attributable to differences in the groups’ composition rather than to any difference in the strategies applied.



The test is then monitored over a pre-set time period to identify any variation in performance between the test and control groups.  Since all other factors are kept constant, superior performance in the test group must be indicative of a superior strategy.  If the test group in our example responds in greater numbers than the control group, we will have proven that colourful envelopes lead to improved response rates and, provided there is no decrease in average sales value, to improved sales. 



With immediate effect, all qualifying candidates who were not mailed during the test should receive the advertisement mailed in a colourful envelope.  This strategy will remain in force until a future test identifies an even more profitable marketing strategy.



Read Full Post »