Metamodel performance measures

UQ Resources Benchmarks
1

Post-prediction Evaluation Metrics

Relative Mean-Squared Error (RelMSE)

The relative mean-squared error is calculated as:

\text{RelMSE} = \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2},

where:

  • y_i is the actual value for the i-th observation,

  • \hat{y}_i is the predicted value for the i-th observation,

  • \bar{y} is the mean of the actual values calculated as \bar{y} = \frac{1}{N}\sum_{i=1}^{N} y_i, and

  • N is the number of data points.

Relative Root-Mean-Squared Error (RelRMSE)

The relative root-mean-squared error is calculated as:

\text{RelRMSE} = \frac{\sqrt{ \sum_{i=1}^{N} (y_i - \hat{y}_i)^2}}{\sqrt{ \sum_{i=1}^{N} (y_i - \bar{y})^2}}.

Relative Mean Absolute Error (RelMAE)

The relative mean absolute error is calculated as:

\text{RelMAE} = \frac{\frac{1}{N} \sum_{i=1}^{N} \left| y_i - \hat{y}_i \right|}{\sqrt{\text{Var}(y)}},

where \text{Var}[y] is the variance of the actual values calculated as \frac{1}{N} \sum_{i=1}^{N} (y_i - \bar{y})^2.

Mean Absolute Percentage Error (MAPE)

The mean absolute percentage error is calculated as:

\text{MAPE} = \frac{1}{N} \sum_{i=1}^{N} \left| \frac{y_i - \hat{y}_i}{y_i} \right| \times 100\%.

Coefficient of Determination (Q2)

The coefficient of determination is calculated as:

\text{Q}^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2}.

A value of 1 indicates perfect prediction, while a value of 0 indicates no predictive power.

Pre-prediction Evaluation Metrics

Relative Cross-Validation Error (RelCVErr)

The relative cross-validation error using leave-one-out (LOO) cross-validation is calculated as:

\text{RelCVErr} = \frac{1}{N} \left[ \frac{ \sum_{i=1}^{N} (y_i - \hat{y}_{i(-i)})^2}{\textrm{Var}[y]} \right],

where \hat{y}_{i(-i)} is the predicted value for the i-th observation obtained by leaving out the i-th data point in the training process.

When using Polynomial Chaos Expansion (PCE) and having results from least-squares minimization, you can directly compute the Leave-One-Out (LOO) error without the need to build N separate metamodels. For more information, see the UQLab user manual – Polynomial chaos expansions, Section 1.4.2.