We are pleased to announce that the application for translation funding for MUCM2 has been awarded by EPSRC. MUCM2 will start on 1st October 2010 and will run for a further 2 years.
MUCM2 will focus on several areas in which we can extend the MUCM technology. In addition to scoping these new areas of research, we plan to consolidate the work we have already done on disseminating our work to the user community.
BACKGROUND TO MUCM
Modelling is a vital part of research and development in almost every sphere of modern life. Those who rely on models to understand complex processes, and for prediction, optimisation and many kinds of decision- and policy-making, increasingly wish to know how much they can trust the model outputs. Uncertainty and inaccuracy in the outputs arises from numerous sources, including error in initial conditions, error in model parameters, imperfect science in the model equations, approximate solutions to model equations and errors in model structure or logic. The nature and magnitudes of these contributory uncertainties are often very difficult to estimate, but it is vital to do so. All the while, for instance, different models produce very different predictions of the magnitude of global warming effects, with no credible error bounds, sceptics can continue to ignore them and pressure groups will seize upon the most pessimistic predictions.
The focus of the MUCM project has been to develop a technology that is capable of addressing all sources of uncertainty in model predictions and to quantify their implications efficiently, even in highly complex models. Furthermore, MUCM seeks to facilitate the management of uncertainty through tools to show how much of the overall uncertainty is due to each contributing source of uncertainty (sensitivity analysis) and to reduce uncertainty by learning from observations of the real-world process (calibration and data assimilation).
MUCM methods are based on two fundamental technical developments. The first is the creation of an emulator for the computer model (usually called the simulator), using a Gaussian process or analogous Bayes linear theory. Emulation based methods are generally orders of magnitude more efficient than traditional Monte Carlo, requiring typically just a few hundreds of model runs, thereby providing very significant productivity gains for the researchers or analysis teams involved. The second development is to model the difference between the simulator output (using ‘best’ input values) and reality, which we call model discrepancy. This is a crucial step in making use of real-world observations, and with emulation allows an integrated approach to uncertainty analysis, sensitivity analysis, calibration and data assimilation – yet model discrepancy is not even acknowledged in traditional approaches to these tasks.
Overall, the project is on track to achieve its principal objectives – delivery of the MUCM technology via the toolkit and case studies, plus developments that explore and push back its boundaries of application.
There is growing international awareness of MUCM and the technology of emulating complex models including collaboration with SAMSI (the Statistics and Applied Mathematics Institute in the Research Triangle, North Carolina), and sessions devoted to this technology at several major international conferences such as the World Congress in Probability and Statistics, the International Society for Bayesian Analysis World Meeting and the International Statistical Institute Annual Meeting.
PLANS FOR MUCM2
We have been awarded a two year extension to the MUCM project (MUCM2). Within MUCM2, in addition to scoping three new areas of research, we plan to consolidate the work we have already done on disseminating our work to the user community. Our plans therefore have four components as described below.
We will study several different kinds of random simulators with a view to scoping promising approaches to tackling the difficulties they post, and particularly to identifying the extent to which different classes of simulators can be addressed in common ways. We will explore techniques for numerous problems in connection with each of the four classes of random simulators as appropriate (micro-simulation, agent models, systems biology models and SFEM).
Models are of course frequently used as an aid to making decisions or policies. In engineering, a simulator of a complex structure such as a motor car engine or a nuclear reactor is used to predict performance of a range of designs, and to select the design with optimal properties. In climate, policy on tackling climate change is necessarily based on models for the consequences of different emissions scenarios. This is another area in which the issues have turned out to be much more complex than we had supposed when we started the MUCM project so in MUCM2 we will address more fully the ways in which models can inform decisions, covering a number of topics including optimisation, coupling models, decisions that expand models, fitness for purpose and risk metrics.
The emulator methods developed in MUCM are predicated on the underlying simulator behaving more or less homogeneously across the input space. That is, the prior belief is that the simulator output should not respond much more dynamically to changes in a given input over some parts of the input space than over others. In particular, there is no prior expectation of sudden shifts, and certainly not discontinuities in the output as the inputs vary smoothly. Emulators can adapt to failure of these prior expectations, but may need large numbers of well-targeted training runs in order to do so. Validation diagnostics are likely to show poor validation even after fitting with a substantial training dataset has seemed to produce accurate emulation, because the predictive uncertainties still do not match appropriately with new validation model runs. Experience in MUCM, particularly with validation but also with some models having clearly heterogeneous behaviour, suggests that this is more of a problem than had been anticipated so we therefore plan to explore ideas for new kinds of emulator which can allow for prior expectations of heterogeneity including discontinuities and heterogeneous variance.
BUILDING THE COMMUNITY
A strategic part of the original MUCM project has been to develop a toolkit and case studies to provide a resource for the community of model users and those working in the field. The toolkit was highly commended by our Mid Term Review panel. We have taken to heart their advice that the toolkit needs a strong editorial control to achieve consistency and quality, and O’Hagan has taken on the role of general toolkit editor. In MUCM2 we plan to further develop the toolkit and case studies, to extend the range of services to the community and to reach out to new user groups. Our objective is to build a real community of users and researchers that will be sustainable beyond the Translation award. We intend to extend the toolkit, deliver short courses, organize video podcasts/seminars and develop the website further.
Further information and webpages for MUCM2 will be made available as this year progresses, so please keep checking the website for new information.