I am new to UQ and working on the analysis of a building structural model under seismic loading. There are 15 random variables (material parameters) in the model and the output is the median seismic response of the structure.
I have already got a data set with the size of 1000 (inputs were obtained by Latin Hypercube Sampling). I trained a PCE model with 800 data samples and used the remaining 200 as a validation set. The errors and the a plot of validation is shown below:
PCE metamodel validation error: 1.0273e-01
PCE metamodel LOO error: 7.6733e-02
I have also run the UQLab code with different training sizes up to 1000, a plot of training size v.s. LOO error is shown below:
As recommended by Professor Sudret (Acceptable LOO Error), a LOO error less than 0.01 is sufficient for obtaining accurate moments or sensitivity indices of the output. So based on the second plot, it seems that I need a lot more data to achieve a LOO error of 0.01.
My questions are:
(1) Does the second plot look normal? The LOO error decreases very slowly with the increase of the training size.
(2) If I cannot achieve a low value of the LOO error, is there anyway to prove that the metamodel is reliable?
Thanks for your question.
I want to share my suggestions. Please follow them:
As Prof. Sudret comprehensively discussed in the Acceptable LOO Error, the LOO error could have bigger values and its dependent on the case study. I think the trend of the LOO error is acceptable in your mentioned figure.
What is the guideline? You could read my discussion on the Goodness-of-fit Tests question.
Besides, based on the suggestions by the UserManual_PCE, please read my answers in the Polynomial chaos expansions question to use the ADAPTIVE options which are implemented in the UQLab.
I hope this helps but please let me know if you have any other questions.
Best regards
Ali
If I may add some thoughts to @ali’s recommendations:
Adaptive PCE can help to get a smaller error, but take care not to choose a too large degree. The number of input dimensions in your problem is quite high. The number of basis functions in your total degree basis should not be too large, at most in the order of several ten thousands for an ED of size 1000. Of course, you can restrict the number of basis functions by setting the q-norm to something smaller than 1. This removes high-order interaction terms and allows you to use a higher degree. You could also try to directly restrict the interaction order to 2 or 3: MetaOpts.TruncOptions.MaxInteraction = 2;
Often, the higher-order interaction terms are not so significant in physics-based models.
Regarding your question how to know whether your metamodel is “reliable”: it depends on what you mean with this, and what you want to use the metamodel for. As @bsudret said, the metamodel typically doesn’t need to be too accurate to yield good estimates for moments and sensitivity indices. LOO error and validation error measure how close the metamodel is to your original model. In your case, it has a validation error of 10%, so it is not that close. In that sense, it is not that “reliable” (for pointwise prediction).
Theorywise, if your model is in L^2_{f_{\bf X}} (i.e. has finite variance under the input distribution), you can approximate it to arbitrary accuracy with polynomials. But in practice, it might be that very many terms, high-order and high-degree, are needed to approximate the model accurately, which requires accordingly many points to identify the coefficients of these terms. Some models are not suited well to being approximated by polynomials, and then the PCE converges only slowly with increasing basis and number of points. The function f(x) = |x| is one example, or discontinuous functions.
So, if you increase the ED and allow for a larger basis, for sure you will get a better approximation - eventually. But it might be that very many points are needed to get a satisfactory approximation.
Regarding your first question: to me it looks like you achieved the maximal accuracy that you can get with the prescribed basis. After 700 points, you don’t gain much accuracy anymore. It might be that a larger basis (using basis adaptivity as recommended by Ali) would help here. The plot itself looks normal.
Good luck, and let us know how you continued with the analysis!
Thank you so much for the detailed explanation. The recommendation of using a truncation option to restrict the interaction order is very useful. Before applying that option, I encountered the problem of running out memory with high degrees.
I would like you and @ali to know that I have rerun the PCE by considering the output as the mean structural response under a group of seismic records, rather than the median response as I described in my original post. In this case, I can obtain a relatively low LOO error (0.01-0.02) with the ED size of 1000.
I guess the reason is that by using the mean response as the output, the input-output relationship became “smoother”.
Thanks again for the kind help. Please let me know if you have any other comments.
Thanks for your information.
As a structural engineer, I am a little bit curious about your quantity of interest.
If it is possible, please let me know that you are working on the global structural parameters such as maximum interstory drift or the local ones like section energy absorption and the plastic hinge deformation?
Happy to hear that we are in the same discipline. I am working on a five story buckling restrained brace frame, the quantity of interest is the maximum interstory drift.
It seems an interesting topic.
Please inform me about your achievements.
Besides, I suggest to consider both maximum interstory drift and residual interstory drift as the QoIs.
Yes exactly, as you say, taking the mean is a smoothing operation and therefore it is much easier to fit than other quantities like e.g. maxima.
It sounds a bit like your model could be a stochastic simulator, since if I understand correctly, the structural response is varying for the same structure properties (inputs) if different seismic records are applied. If you are only interested in the mean response, a PCE is enough. But if you are interested in more properties of the response, e.g. quantiles or the distribution, you might want to consider methods for stochastic simulators.
