The need for more accurate and realistic project planning for MOD procurement was clear from the research conducted by former Chief of Defence Materiel, Bernard Gray. In his paper, Review of Acquisition for the Secretary of State for Defence, published in October 2009, 40% of the 40 major projects investigated, cost more than expected and also delivered 80% later than the first estimates predicted, resulting in a £35bn cost overrun. His verdict on the MOD equipment programme was that it was “substantially overheated” and “unaffordable on any likely projection of future budgets”. Nearly 10 years after these predictions were first made and his concluding statements still hold true today with a potential £3.2bn understatement being quoted by CAAS against the latest Equipment Plan (2017-2027). This figure is included in the total risk facing the MOD which is estimated to rise to as high as £20.8bn according to the NAO, 11.6% of the total equipment and support budget over the 10 year period.
All of the estimates and inputs of project planning can be affected by human behavior, either consciously or subconsciously, which can reduce the validity and realism of the final outputs. This can be seen in both tender proposals submitted by industry suppliers and contained within MOD plans. Therefore to accurately estimate the cost, benefit and delivery timescales of a new project an accurate and empirically based optimism bias adjustment needs to be included. This adjustment accounts for all known risks that may affect the delivery of the project and is imperative for accurate cost forecasting. Without the application of an appropriate optimism bias, project schedules and project budgets can be underestimated and therefore making it much harder to allocate the total budget effectively. Governance should be put in place to counter optimism bias. According to the Northern Ireland Department of Finance, to allow a valid and robust assessment of project plans the optimism bias should be based on data from past projects or similar projects elsewhere, and adjusted for the unique characteristics of the project in hand. All estimates should be scrutinised for realism and assurances obtained that the estimates have valid justifications where possible.
The current approach to adjusting optimism bias within the MOD originates from the HM Treasury Green Book guidelines. The research fueling the HM Treasury Green Book’s optimism bias adjustments was conducted in 2002 by Mott Macdonald within their paper Review of Large Public Procurement in the UK. However the data set used for this was across all public sector departments, not specifically tailored to MOD projects. Only 3 of the 50 projects included within the final data set used for the final optimism bias estimates are directly related to MOD work, therefore making the majority of the estimates inappropriate for the highly-complex portfolio of MOD projects.
In addition, their work is now 16 years out of date. There have been a number of significant changes to procurement and project management techniques since 2002, like the emergence of agile deliveries and the digital revolution, both resulting in efficiencies in project delivery and customer insight capabilities. The Green Book guidelines suggesting the use of a constant optimism bias has been a useful interim measure but now with the emergence of big data in recent years, it should be reinvented to match the dynamic environment of the MOD. If the National Audit Office believe there is to be an estimated £20.8bn overspend in the latest equipment plan then is quite apparent that the process in place for generating and scrutinising project plans is not working and not suitable for the nature of MOD procurement.
Therefore our recommendation is to no longer treat this seemingly insignificant adjustment to project plans as a constant, but to continually adjust a “smart” optimism bias generated from past project data. With the MOD’s implementation of release 2 of the Contracts, Purchasing & Finance “e-procurement” solution in December 2016, project data is much more readily available than the Mott Macdonald’s optimism bias averages made in 2002. From CP&F and the standing up of the Project Controls function in DE&S, there will be a wealth of project data collected and this is essential in being used as the foundations for the basis of estimates and validating similar projects. Detailed analysis will then need to be conducted to generate an optimism bias for new projects dependent on variables like project type, size and teams involved, each with a fixed weighting. The MOD need to utilise the data available to them to provide a more rigorous review and assessment and inject a level of realism into project plans to counter any optimism bias.
Mitchell Hooker and Simon Harrison, RINA
