GPLayer's prediction seems to be too confident
Closed this issue · 3 comments
Looking at the "Hybrid Deep GP models: ..." tutorial, the GPLayer's prediction seems to be too confident, i.e. its uncertainty estimate (95% confidence level) does not cover the training data spread. Its prediction accuracy (mean), however, is very good, nearly identical to that of the neural network model obtained by removing the GPLayer.
When I replace the GPLayer with a TFP DenseVariational layer using Gaussin priors, the prediction accuracy is not as good. However, importantly, the uncertainty estimate is very good, covering the training data spread well.
Without good uncertainty estimate, the GPLayer seems to add little value over the neural network model, which already provides good prediction accuracy.
Hi @dtchang, I had another look at the notebook and I think the fit is actually pretty good. In this PR, I updated the notebook so that it plots the mean and variance of y instead of f. In the figure below, you can see that the fit now encloses the data. You can also observe the increased uncertainty on f in the far left and right of the domain.
I still can't find the updated notebook in the GitHub. But, I think this issue should be re-opened because it is not a plotting problem, but a prediction problem. f is the prediction, whereas y is the target. So, it seems to me the plot should be based on the mean and variance of f, not y, as in the published tutorial. BTW, the issue of poor uncertainty estimation exists not just in this tutorial, but also in other tutorials (e.g., "Introduction to GPflux" and "Why GPflux is a modern (deep) GP library").