Dear cummunity,
As the title, can we use Bayesian method to estimate the hyper-parameter of Gaussian Process regression, in UQLab?
I used GP for regression task, however, overfitting seems occur. So can we use Bayesian method in UQLab to eliminate the overfitting?
Any advice?
Hi @GPLai,
The hyperparameters of the Gaussian Process (GP) model in UQLab are estimated using either the maximum-likelihood estimation or cross-validation estimation. The estimated hyperparameters are then plugged in into the model and used as is for prediction (i.e., they are plug-in estimators). As far as I know, UQLab does not support a Bayesian estimation of these hyperparameters out-of-the box.
However, with some efforts you might be able to set up your own Bayesian framework to estimate them.
According to the Kriging user manual Section 2.9, you can set up a Kriging predictor manually given all the hyperparameters (a custom Kriging-based model). While custom based Kriging models do not contain the objective function evaluation, it contains the correlation and observation matrix for the specified trend and correlation function (see Reference table at the back of the manual). I think, along with the specified GP variance, these are the ingredients you need to compute the likelihood (or the logarithm of it). If you put priors on the hyperparameters and specify some likelihood (for example, see Eq. 1.39 in the user manual), you can then compute posterior of the hyper-parameters (say, by simulation).
I’ve never tried this myself, so I’m not sure how it’s going to go; If you’re interested you might want to give it a try and let us know how it goes! 
I hope this helps!
PS: Out of curiousity, how did you analyze the overfitting in your GP model?
1 Like