Christine Shoemaker came and talked about a method she has that essentially is one way to answer the question I posed recently "Where do Likelihoods Come From?" For her problems the model is a very expensive computational thing, like time-stepping a PDE for the concentration of pollutants underground. The application I'm interested in is likelihood functions in MCMC, though most of her talk was about optimization of mean sum of squared errors, she did talk towards the end about applications to MCMC.

Her technique is to evaluate the model prediction using the expensive simulation function, let's call it $F(x)$ where $x$ is the set of parameters, then compare it to data $D$ to get a posterior evaluation $P(D|x)P(x)$. You do this for some set of points, and construct an approximate function $P^*(D|x)P(x)$ using Radial Basis Functions. Then you identify a high probability region of the parameter space $x$ using optimization techniques on this $P^*$ function, and you jump around evaluating $P(D|x)P(x)$ for $x$ values in this high probability region to improve your $P^*$ RBF approximation until you're happy (you're not doing MCMC yet, you're just trying to get a good interpolation of the posterior density) once you are happy with the $P^*$ approximation you can start at some point in the high probability manifold, and rapidly sample $P^*(D|x) P(x)$ in detail using MCMC without running an expensive numerical simulation. Some of these numerical simulations might take 30 minutes to 10 hours for each simulation, so it's clearly impossible to do MCMC very well on the raw numerical model.

This will give you the approximate distribution of the posterior parameter values of interest, and finally if you're interested in prediction you can sample from these posterior parameter values, and run your expensive model a few times on the high probability parameter values to get predictions about the future.

She found speedups of around 20 to 60 times fewer evaluations of the full numerical simulation for her method, and I would say this is one of the most important talks I've heard in a long time since it makes it practical to do Bayesian MCMC on the kinds of models that Civil and Environmental engineers are really interested in, things like timeseries for pollution transport, flooding, building earthquake response, etc. Since the RBF function is easily differentiable it would also be very easy to do HMC on such an approximation, whereas not so much for raw evaluations of the numerical simulations.

One Response leave one →
1. May 2, 2013

I'm very tempted to apply this idea to the falling ball problem as an example but I really need to focus on finishing up my existing research and graduating soon.