First of all I would like to say thanks for the freely available UQLab-Framework! I have been trying for a while to create sparse PCEs for different output variables from FE calculations using random fields and different material models. It works very well for some material models, but not so well for other material models so far.
However, for all the calculations, I have noticed a large discrepancy between the leave-one-out error and the validation error. In most cases a LOO error was reported which was one or more powers of 10 smaller. I observed this using linear-elastic calculations as well as non-linear calculations.
Based on the equations for both error measures in the user manual, I can’t explain the differences since both equations do almost the same normalization and I always chose my number of validation samples to be at least 200-500.
Since the adaptive choice of p and q are based on the LOO error, I have often manually experimented to find out with which p and q I get the lowest validation error for this reason. This gives me different “optimal” values for p and q than if I let p and q be automatically selected adaptively.
Is this difference between the error measures normal, or am I doing something fundamentally wrong?
I would be very happy about your comments. Thank you in advance!
Esther
Sorry for taking so long to answer. Great to heat that you are finding UQLab useful!
Yes, PCE does not work for all models equally well. Especially if models are discontinuous of non-smooth (e.g., scalar models resulting from a maximum operation), PCE is often not able to approximate the model very well unless very high degrees are used (and that again might require infeasibly many model evaluations).
To your question about validation errror and LOO: In principle, LOO is supposed to approximate the validation error well. This might not be the case if your training data set (the experimental design) is very small. Another observation that we have made is that for some reason, the LOO provided by the method OMP is not reliable at all. Were you using OMP by any chance? If yes, consider using another method such as LARS or SP.
If you are choosing p and q based on the resulting validation error, you are using information from the validation set to compute the coefficients. After that, the validation error that you compute will not be the true validation error anymore! It is normal that the p and q chosen based on LOO are not the same as the ones that would give the smallest validation error, but they should at least be sort of close.
