Understanding leave one out error and modified leave one out error in PCE

All About UQ UQ Discussion Forum
,
1

Hello,
I have been using UQLab for modelling and predicting the outputs of a filter in the frequency domain.
I have used the Kriging and Polynomial Chaos Expansion for the same. Kriging works just fine, but I was trying out the simulations with PCE so as to compare the two models.
What I am having trouble understanding is that on printing the PCE report, we get two errors, Leave one out error and a Modified Leave one-out error, can someone explain the difference between the two?

PCE_vs_Krg
PCE_vs_Krg1920×926 13.3 KB

The error I get from the report is: Leave-one-out error: 4.6383164e-02
Modified leave-one-out error: 1.4639779e-01

Now, which error should I use to actually compare to some other model such as the kriging?

Thank You

2

Dear @ayush

To give you an overview:

  • The leave-one-out (LOO) cross-validation error for Kriging is defined in this manual by Eq.1.61
  • The LOO error for PCE is defined in this manual by Eq.1.19
  • The modified LOO error for PCE is defined in this manual by Eq.1.22

Eq. 1.61 (Kriging manual) and Eq. 1.19 (PCE manual) are identical. However, if you fit your PCE model using least-squares, Eq. 1.20 (PCE manual) is used to (significantly) facilitate the computation of the LOO error. But that is mainly an implementation detail.

The modified LOO error is equal to the LOO error times a correction factor that depends on the number of regressors of your model and the number of training samples. It’s purpose is to account for overfitting if the experimental design is small. From its definition in Eq. 1.22-1.23 (PCE manual) it becomes apparent that it is strictly larger than the non-modified LOO error.

Hope this helps you getting started.

Best regards
Styfen