Monday

Tag Archives: Averages

Averaging the Power of Portfolios

Posted on July 8, 2012 by | 4 comments

The interaction between dependent or connected risk and independent risk is interesting and will significantly change the overall probability of success or failure of an endeavour or organisation.

As discussed in my last post on ‘The Flaw of Averages’  using a single average value for an uncertainty is a recipe for disaster. But there is a difference between averaging, connecting and combining uncertainties (or risk).

Adding risk

Where risk events are connected, the ability to model and appreciate the effect of the risk events interacting with each other is difficult. In ‘The Flaw of Averages’ Sam Shaw uses the simile of wobbling a step ladder to determine the uncertainty of how safe the ladder is to climb. You can test the stability of one ladder by giving it a good ‘wobble’. However, if you are trying to determine the stability of a plank between two stepladders doubling the information from wobbling just one is not a lot of help. Far more sophisticated modelling is needed and even then you cannot be certain the full set of potential interactions is correctly combined in the model. The more complex the interactions between uncertainties, the less accurate the predictive model.

However, when the risks or uncertainties are independent, combining the risks through the creation of a portfolio of uncertainties reduces the overall uncertainty quite dramatically.

The effect of portfolios

Consider a totally unbiased dice, any one throw can end up anywhere and every value between 1 & 6 has an equal probability of being achieved. The more throws, the more even the results for each possibility and consequently there is no possibility of determining the outcome!

The distribution after 10, 100 and 1000 throws.

As the number of throws increase, the early distortions apparent after 10 throws smooth out and after 1000 throws the probabilities are almost equal.

However, combine two dice and total the score results in a very different outcome. Whilst it is possible to throw any value between 2 & 12, the probability of achieving a number nearer the middle of the range is much higher than the probability of achieving a 2 or a 12. The potential range of outcomes starts to approximate a ‘normal distribution curve’ (or a bell curve). The reason for this is there is only one combination of numbers that will produce a 2 or a 12; there are significantly more combinations that can make 7.

The more dice you add to the ‘throw’, the closer the curve becomes to a ‘normal distribution’ (or bell curve), which is normally what you expect/get, which is the origin of the name!

The consequence of this phenomenon is to demonstrate that the creation of a portfolio of projects will have the effect of generating a normal distribution curve for the outcome of the overall portfolio, which makes the process of portfolio management a more certain undertaking than the management of the individual projects within the portfolio. The overall uncertainty is less than the individual uncertainties……

Each project carries its level of uncertainty and has a probability of succeeding off-set by a probability of failing (see Stakeholder Risk Tolerance) but as more projects are added the probability of the overall portfolio performing more or less as expected increases, provided each of the uncertainties are independent! This effect is known as the Central Limit Theorem.

One important effect of the Central Limit Theorem is the size if the contingency needed to achieve a desired level of safety for a portfolio of projects is much smaller than the sum of the contingencies needed to achieve the same level of ‘safety’ in each of the individual projects. Risk management is a project centric process; contingency management is better managed at the portfolio level. Not only is the overall uncertainty reduced, but the portfolio manager can offset losses in one project against gains in another.

Whist this theorem is statistically valuable, the nature of most organisations constrains the potential benefit. From a statistical perspective diversity is the key; this is why most conservative investment portfolios are diversified. However, project portfolios tend to be concentrated in the area of expertise of the organisation which removes some of the randomness needed for the Central Limit Theorem to have its full effect.

It is also important to remember that whilst creating a portfolio will reduce uncertainty, no portfolio can remove all uncertainty.

In addition to the residual risk of failure inherent in every project, there is always the possibility of a ‘black swan’ lurking in the future. Originally conceptualized by philosopher Karl Popper and refined by N. N. Taleb, a ‘black swan’ is a risk event that has never occurred before, if it did occur would have and extreme impact and is easy to explain after the event, but is culturally impossible to predict in advance (ie, the event could be foreseen if someone is asked to think about it but it is nearly impossible to think the thought for a compelling reason). For more on black swans see our blog post  and White Paper.

The Law of Averages

The Central Limit Theorem is closely aligned to The Law of Averages. The Law of Averages states that if you repeatedly take the average of the same type of uncertain number the average of the samples will converge to a single result, the true average of the uncertain number. However, as the ‘flaw of averages’ has demonstrated, this does not mean you can replace every uncertainty with an average value and some uncertain numbers never converge.

Summary

Both the Law of Averages and Central Limit Theorem are useful concepts; they are the statistical equivalent of the adage “don’t put all your eggs in one basket”. When you create a portfolio of projects, the average probability of any one project succeeding or failing remains the same as if the project was excluded from the portfolio, but the risk of portfolio suffering an overall failure becomes less as the number of projects included in the portfolio increases.

However, unlike physical laws such as gravity, these laws are not immutable – drop an apple within the earths gravitational pull and it will fall; create a portfolio and there is always a low probability that the results will not conform to normal expectations!

Certainly the probability of a portfolio of projects ‘failing’ is lower then the average probability of each project failing but a reduced level of risk still leaves a residual level of risk.

4 Comments

Posted in Risk, Value

Tagged , , , , , ,

The flaw of averages

Posted on June 22, 2012 by | 2 comments

The flaw of averages defined in a book of the same name by Sam L. Savage, states in effect, any plan based on average assumptions is wrong on average! http://www.flawofaverages.com/

However, every duration estimate, cost estimate, risk impact and other estimate our project plans are based on an ‘average’ or ‘expected value’ derived from past experience. And as naturalist Stephen Jay Gould commented, our culture encodes a strong bias either to neglect or ignore variation. We tend to focus instead on measures of central tendency, and as a result we make some terrible mistakes, often with considerable practical import.

The flaw of averages ensures plans based on a single average value that describes an uncertainty will be behind schedule and over budget! A typical example from the book looks at a stocking problem – the business is planning to import short shelf life exotic fruits with a high profit margin, the marketing team have analysed the market and developed a profile of likely sales. The boss looks at the distribution and demands a single figure. All the marketing team can do is take the ‘average’ expected sales and decide 500 cases per month are the most likely level of sales. Based on a profit of $100 per case the boss predicts a net profit of $50,000 per month. However, this is a very optimistic estimate, if less than 500 cases are sold, the fruits will spoil with losses of $50 per case, if more than 500 cases are required the cost of airfreighting extra cases is $150 per case resulting in a loss of $50 or the sales have to be foregone with a risk of losing the customer.

The highest possible monthly profit is $50,000 – if more or less are sold the profit reduces. On average each month more or less than 500 cases will be sold, resulting in returns lower than the estimated $50,000. The only time the predicted profit will be realised in the occasional month when exactly 500 cases are sold.

Even if the company decides not to airfreight additional cases on average the monthly profit will be less than $50,000. Without airfreight, for roughly half the time demand will exceed 500 cases but with no additional stock, profit is capped at $50,000. For the other months, sales will be less than 500 and there will be spoilage costs. Meaning on average, the monthly profit will be less than predicted!

The average is correct, the way the manger is using the average is the ‘flaw’. The same problem shown in the cartoon above, ‘on average’ the pond is only 1 meter (3ft) deep! But averages are rarely what is needed for prudent management.

To properly analyse the projected profits more in-depth analysis is needed, using techniques such as Monte Carlo analysis with the variability of sales being represented by the input probability distribution, the costs and income expected modelled in the tool and the resulting profits predicted in the output probability distribution.

The challenge is getting valid data to model. Projects are by definition ‘unique endeavours’ which means there is no pool of directly valid data; this problem is discussed in our paper The Meaning of Risk in an Uncertain World . When managing project uncertainties our basic data is uncertain!

Recognising this simple fact is a major step towards better project management. To quote George Box (Stamford University) ‘All models are wrong, some models are useful’. No model should be taken as correct, this includes schedules, cost plans, profit predictions, risk simulations and every other predictive model we use! They are never complete representations of exactly what will occur, but a successful model will tell you things you did not tell it to tell you (Jerry P. Brashear).

Building a successful model such as a useful schedule (useful schedules are useful because they are used) should go through the five stages defined by Donald Knuth:
1. Decide what you want the model to do
2. Decide how to build the model
3. Build the model
4. Debug the model
5. Trash stages 1 through 4 now you know what you really want.

And to get a large model to work, you must start with a small model that works, not a large model that does not work. If you want to understand flight what is more useful, a large highly detailed model of a Boeing Jumbo jet built out of Lego blocks that cannot fly or a simple paper aeroplane that does?

The complex Lego model may be visually impressive but is likely to be less useful in understanding a dynamic process such as flight.

The same is likely to be true for most dynamic project models. Edward Tufte says ‘Clear and precise seeing becomes as one with clear and precise thinking’, and John W. Tukey adds ‘It is far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made precise.’ It is dumb to be too smart!

These concepts are consistent with the PMBOK® Guide idea of ‘progressive elaboration’ and are embedded in the scheduling technique called ‘Schedule Density’ where the initial schedule is developed at ‘Low Density’ and additional detail added as needed (see more on Schedule Density).

The message from this blog is building a useful model is a skilled art, regardless of the subject being modelled (time, cost, risk). A good start is to keep the model simple, if you don’t understand how the model works how will you will be able to judge what it shows you? The model is never the truth; at best it is a useful! And its usefulness will be severely reduced if you rely on averages such as single point estimates without at least using some probability analysis. Melding the need for precision with probabilistic assessments are discussed in our paper Why Critical Path Scheduling (CPM) is Wildly Optimistic.

Whilst this post has focused on one dimension of uncertainty (time and schedule), the principles can be applied to any area of uncertainty.

2 Comments

Posted in Risk, Scheduling

Tagged , , , , ,