--- title: "Prediction performance metrics" author: "Alexandre M.J.-C. Wadoux" output: rmarkdown::html_vignette bibliography: references.bib link-citations: true vignette: > %\VignetteIndexEntry{Prediction performance metrics} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 5 ) ``` ## 1. Why use several prediction metrics? Continuous predictions may be generated using machine-learning algorithms, statistical models, geostatistical methods, or process-based models. `modelskill` provides tools to evaluate the agreement between paired observations and predictions using a range of complementary performance metrics. Predictive performance is inherently multidimensional. A model may exhibit low overall error while retaining systematic bias, reproduce the observed pattern while consistently over- or underpredicting, or achieve good aggregate accuracy while failing to capture the observed variability. Accordingly, model evaluation should generally rely on a combination of complementary metrics rather than a single summary statistic, so that different aspects of predictive performance can be assessed explicitly. Performance metrics should be computed using independent validation data whenever possible, or using an appropriate resampling or cross-validation procedure when an independent validation dataset is not available. ## 2. Example data We create three deliberately different prediction models. ```{r data} library(modelskill) set.seed(123) n <- 100 obs <- seq(0, 10, length.out = n) + rnorm(n, sd = 1) models <- list( Good = obs + rnorm(n, sd = 0.5), # Constant positive offset Biased = obs + 1, # Larger random errors Noisy = obs + rnorm(n, sd = 2) ) ``` The models have simple characteristics: * `Good` has relatively small random errors; * `Biased` systematically overpredicts by one unit; * `Noisy` has larger random errors. ## 3. Start by looking at the predictions Before calculating summary statistics, it is useful to inspect observed versus predicted values. ```{r scatter} plot_data <- data.frame( observed = rep(obs, times = length(models)), predicted = unlist(models, use.names = FALSE), model = rep(names(models), each = length(obs)) ) ggplot2::ggplot( plot_data, ggplot2::aes( x = observed, y = predicted, colour = model ) ) + ggplot2::geom_abline( slope = 1, intercept = 0, linetype = "dashed" ) + ggplot2::geom_point( alpha = 0.7, size = 1.5 ) + ggplot2::coord_equal() + ggplot2::theme_classic() + ggplot2::labs( x = "Observed", y = "Predicted", colour = "Model" ) ``` The dashed 1:1 line represents perfect agreement. `Good` remains relatively close to this line, `Biased` is systematically shifted upward, and `Noisy` is more dispersed. # 4. Calculate individual metrics Every prediction metric in `modelskill` can be calculated separately. This is useful when only a few specific aspects of performance are required. All individual functions use the same basic argument order: ```r metric(obs, pred) ``` where `obs` is the vector of observations and `pred` is the corresponding prediction vector. ## 4.1 RMSE: overall error magnitude The **root mean squared error (RMSE)** measures the overall magnitude of prediction errors: \[ \mathrm{RMSE} = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} (obs_i-pred_i)^2 }. \] Calculate RMSE for the `Good` model: ```{r rmse-good} rmse( obs, models$Good ) ``` and compare it with the `Noisy` model: ```{r rmse-compare} rmse( obs, models$Noisy ) ``` [`rmse()`](../reference/rmse.html) is non-negative and has the same units as the response variable. Zero indicates perfect predictions and smaller values indicate smaller errors. Because errors are squared before averaging, relatively large errors have more influence on RMSE. It is therefore useful to interpret RMSE together with a less outlier-sensitive statistic such as [`mae()`](../reference/mae.html). ```{r rmse-mae} rmse(obs, models$Noisy) mae(obs, models$Noisy) ``` ## 4.2 Bias: systematic overprediction or underprediction `modelskill` defines prediction error as \[ e_i = obs_i-pred_i. \] The [`bias()`](../reference/bias.html) function calculates the mean signed error: \[ \mathrm{ME} = \frac{1}{n} \sum_{i=1}^{n} (obs_i-pred_i). \] For the deliberately biased model: ```{r bias-example} bias( obs, models$Biased ) ``` Because `Biased = obs + 1`, its mean error is -1. Under the `modelskill` convention: * **negative bias** means overprediction; * **positive bias** means underprediction; * **zero bias** means no average systematic error. A bias close to zero does not necessarily mean that predictions are accurate. Positive and negative errors can cancel. Bias should therefore be interpreted with an unsigned error metric such as MAE or RMSE. # 5. A particularly important distinction: `r2()` versus `R2()` `modelskill` contains two statistics whose names differ only by capitalization, but they measure different things. ### 5.1 Lowercase `r2()` [`r2()`](../reference/r2.html) is **squared Pearson correlation**: \[ r^2 = \left[ \frac{ \sum_{i=1}^{n} (obs_i-\bar{obs})(pred_i-\bar{pred}) }{ \sqrt{ \sum_{i=1}^{n}(obs_i-\bar{obs})^2 \sum_{i=1}^{n}(pred_i-\bar{pred})^2 } } \right]^2. \] It measures the strength of linear association. ### 5.2 Uppercase `R2()` [`R2()`](../reference/efficiency_r2.html) is the **model-efficiency coefficient**: \[ R^2 = 1- \frac{ \sum_{i=1}^{n}(obs_i-pred_i)^2 }{ \sum_{i=1}^{n}(obs_i-\bar{obs})^2 }. \] It compares the model with using the observed mean as a constant prediction. This distinction becomes obvious with the `Biased` model: ```{r r2-versus-R2} r2( obs, models$Biased ) R2( obs, models$Biased ) ``` The biased predictions are exactly one unit above the observations. Their linear association is therefore perfect, so lowercase `r2()` is 1. However, the predictions are not perfectly accurate. Uppercase `R2()` detects their prediction error and is therefore below 1. This illustrates why correlation and squared correlation should not be used alone as measures of prediction accuracy. > **Remember:** lowercase `r2()` is squared Pearson correlation. Uppercase > `R2()` is model efficiency. In `modelskill`, uppercase `R2()`, [`nse()`](../reference/nse.html), and [`mec()`](../reference/mec.html) are aliases for exactly the same statistic: ```{r efficiency-aliases} c( R2 = R2(obs, models$Good), NSE = nse(obs, models$Good), MEC = mec(obs, models$Good) ) ``` For this statistic: * **1** is perfect prediction; * **0** means that predicting the observed mean performs equally well; * **negative values** mean that predicting the observed mean would perform better. The distinction between \(r^2\) and \(R^2\) (also referred to as NSE or MEC) has been highlighted on many occasions [e.g., @legates1999; @wadoux2022]. It is also discussed on the [Wikipedia page on the coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination), which notes that \(r^2\) quantifies the strength of the linear relationship between observed and predicted values, whereas evaluation of predictive goodness-of-fit should concern the specific 1:1 relationship, $$ obs = 1 \times pred + 0. $$ A high \(r^2\) can therefore occur even when predictions are systematically biased or otherwise depart substantially from the 1:1 line. ## 5.3 Correlation is not agreement The same issue can be illustrated by comparing Pearson correlation with Lin's concordance correlation coefficient. ```{r correlation-versus-ccc} correlation( obs, models$Biased ) ccc( obs, models$Biased ) ``` [`correlation()`](../reference/correlation.html) measures linear association. It is insensitive to a constant shift. [`ccc()`](../reference/ccc.html) measures concordance and is reduced when predictions differ from observations in their mean or variability. It therefore measures agreement more directly than Pearson correlation. # 6. Compare several models at once When several models are being evaluated, [`model_metrics()`](../reference/model_metrics.html) calculates the main prediction metrics together. ```{r metrics} model_metrics( models, obs, digits = 3 ) ``` The default output contains complementary information about: * systematic error; * average error magnitude; * centred error; * association; * model efficiency; * variability; * concordance. For the example data, no single column should be used to identify the "best" model without considering what aspect of prediction quality matters for the application. # 7. Categories of prediction performance metrics The following classification is intended as a practical guide. Some metrics are mathematically related and should not be interpreted as independent evidence. | Category | `modelskill` functions | Main question | |:--|:--|:--| | **Systematic error** | [`bias()`](../reference/bias.html), [`mpe()`](../reference/mpe.html) | Are predictions systematically too high or too low? | | **Error magnitude** | [`mae()`](../reference/mae.html), [`mdae()`](../reference/mdae.html), [`mse()`](../reference/mse.html), [`rmse()`](../reference/rmse.html) | How large are the prediction errors? | | **Centred error and spread** | [`crmse()`](../reference/crmse.html), [`sep()`](../reference/sep.html), [`sd_ratio()`](../reference/sd_ratio.html) | How much disagreement remains after bias is removed, and is variability reproduced? | | **Association** | [`correlation()`](../reference/correlation.html), [`r2()`](../reference/r2.html) | Do predictions reproduce the observed pattern? | | **Agreement** | [`ccc()`](../reference/ccc.html), [`willmott_d()`](../reference/willmott_d.html) | How closely do predicted values agree with observations? | | **Efficiency and benchmark performance** | [`R2()`](../reference/efficiency_r2.html), [`nse()`](../reference/nse.html), [`mec()`](../reference/mec.html), [`rae()`](../reference/rae.html), [`kge()`](../reference/kge.html) | How well does the model perform relative to a benchmark or multiple performance components? | | **Scaled and relative error** | [`nrmse()`](../reference/nrmse.html), [`rrmse()`](../reference/rrmse.html), [`rpd()`](../reference/rpd.html), [`rpiq()`](../reference/rpiq.html), [`rer()`](../reference/rer.html) | How large is error relative to the scale or variability of the observations? | | **Relative and transformed losses** | [`mape()`](../reference/mape.html), [`smape()`](../reference/smape.html), [`msle()`](../reference/msle.html), [`rmsle()`](../reference/rmsle.html) | How large is error on a relative or logarithmic scale? | | **Quantile loss** | [`pinball_loss()`](../reference/pinball_loss.html) | How accurate is a prediction for a specified quantile? | There is no universally best metric. The appropriate combination depends on the scientific objective and the consequences of different types of prediction error. # 8. Core metrics The default output of `model_metrics()` contains the principal metrics used for continuous prediction evaluation. For the retained observation-prediction pairs, let \(o_i\) denote observations, \(p_i\) predictions, and \(n\) the number of pairs. Let \(\bar{o}\) and \(\bar{p}\) denote their means. | Metric / function | Equation | Ideal | Simple interpretation | Reference | |:--|:--|:--|:--|:--| | **ME / bias** [`bias()`](../reference/bias.html) | \(\mathrm{ME}=\frac{1}{n}\sum_i(o_i-p_i)\) | 0 | Average signed error. Negative = overprediction; positive = underprediction. | [@legates1999] | | **MAE** [`mae()`](../reference/mae.html) | \(\mathrm{MAE}=\frac{1}{n}\sum_i\lvert o_i-p_i\rvert\) | 0 | Average absolute error in response units. | [@willmottmatsuura2005] | | **MSE** [`mse()`](../reference/mse.html) | \(\mathrm{MSE}=\frac{1}{n}\sum_i(o_i-p_i)^2\) | 0 | Average squared error; gives more weight to large errors. | [@hodson2022] | | **RMSE** [`rmse()`](../reference/rmse.html) | \(\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_i(o_i-p_i)^2}\) | 0 | Overall error magnitude in response units; sensitive to large errors. | [@hodson2022] | | **NRMSE** [`nrmse()`](../reference/nrmse.html) | \(\mathrm{NRMSE}=\mathrm{RMSE}/s_o\) | 0 | RMSE relative to the observed standard deviation. | [@taylor2001] | | **cRMSE** [`crmse()`](../reference/crmse.html) | \(\mathrm{cRMSE}=\sqrt{\frac{1}{n}\sum_i[(o_i-p_i)-\mathrm{ME}]^2}\) | 0 | Error remaining after mean bias is removed. | [@taylor2001] | | **Pearson correlation** [`correlation()`](../reference/correlation.html) | \(r=\frac{\sum_i(o_i-\bar{o})(p_i-\bar{p})}{\sqrt{\sum_i(o_i-\bar{o})^2\sum_i(p_i-\bar{p})^2}}\) | 1 | Strength and direction of linear association; not agreement. | [@legates1999] | | **Squared Pearson correlation** [`r2()`](../reference/r2.html) | \(r^2 = r^2\) | 1 | Strength of linear association without its sign; not agreement. | [@legates1999] | | **Model efficiency \(R^2\)** [`R2()`](../reference/efficiency_r2.html) | \(R^2 = 1-\frac{\sum_i(o_i-p_i)^2}{\sum_i(o_i-\bar{o})^2}\) | 1 | 1 = perfect; 0 = no better than predicting the observed mean; negative = worse than predicting the observed mean. | [@nash1970; @janssen1995] | | **SD ratio** [`sd_ratio()`](../reference/sd_ratio.html) | \(\mathrm{SD\ ratio} = \frac{s_p}{s_o}\) | 1 | Below 1 = too little variability; above 1 = too much variability. | [@taylor2001] | | **CCC** [`ccc()`](../reference/ccc.html) | \(\mathrm{CCC} = \frac{2c_{op}}{v_o+v_p+(\bar{o}-\bar{p})^2}\) | 1 | Agreement in correlation, mean, and variability. | [@lin1989] | | **Bias correction factor \(C_b\)** [`model_metrics()`](../reference/model_metrics.html) | \(C_b = \frac{2\sqrt{v_ov_p}}{v_o+v_p+(\bar{o}-\bar{p})^2}\) | 1 | Agreement in mean and variability, without correlation. | [@lin1989] | Here \(s_o\) and \(s_p\) are the sample standard deviations of observations and predictions. For CCC, \(v_o\), \(v_p\), and \(c_{op}\) are the corresponding population variances and covariance used by the package. The ranges and special cases of each metric are described in its linked reference page. # 9. Extended metrics Additional metrics are available when a more specialised evaluation is required. ```{r extended} model_metrics( models, obs, extended = TRUE, digits = 3 ) ``` The extended output adds robust error measures, scale-normalised metrics, percentage errors, agreement measures, and KGE (2009). These metrics should be selected because they answer a relevant scientific question, not simply because they are available. For example, the median absolute error is less affected by a few unusually large errors than RMSE: ```{r mdae-example} mdae( obs, models$Noisy ) ``` KGE (2009), the original Kling-Gupta efficiency formulation, combines correlation, variability, and mean agreement: ```{r kge-example} kge( obs, models$Good ) ``` ## 9.1 Extended metric reference | Metric / function | Equation | Ideal | Simple interpretation | Reference | |:--|:--|:--|:--|:--| | **MdAE** [`mdae()`](../reference/mdae.html) | \(\mathrm{MdAE} = \mathrm{median}_i\lvert o_i-p_i\rvert\) | 0 | Typical absolute error; relatively robust to large errors. | [@hyndman2006] | | **RPD** [`rpd()`](../reference/rpd.html) | \(\mathrm{RPD} = s_o/\mathrm{RMSE}\) | Larger | Error relative to observed standard deviation. | [@bellonmaurel2010] | | **RPIQ** [`rpiq()`](../reference/rpiq.html) | \(\mathrm{RPIQ} = \mathrm{IQR}(o)/\mathrm{RMSE}\) | Larger | Error relative to the observed interquartile range. | [@bellonmaurel2010] | | **SEP** [`sep()`](../reference/sep.html) | \(\mathrm{SEP} = \sqrt{\frac{1}{n-1}\sum_i[(o_i-p_i)-\mathrm{ME}]^2}\) | 0 | Sample SD of prediction errors after removing bias. | [@bellonmaurel2010] | | **RER** [`rer()`](../reference/rer.html) | \(\mathrm{RER} = \frac{\max(o)-\min(o)}{\mathrm{RMSE}}\) | Larger | Error relative to observed range; sensitive to extreme values. | [@bellonmaurel2010] | | **MAPE** [`mape()`](../reference/mape.html) | \(\mathrm{MAPE} = \frac{100}{n}\sum_i\left\lvert\frac{o_i-p_i}{o_i}\right\rvert\) | 0 | Mean absolute percentage error. Undefined when an observation is zero. | [@hyndman2006] | | **MPE** [`mpe()`](../reference/mpe.html) | \(\mathrm{MPE} = \frac{100}{n}\sum_i\frac{o_i-p_i}{o_i}\) | 0 | Signed relative bias in percent. Undefined when an observation is zero. | [@hyndman2006] | | **sMAPE** [`smape()`](../reference/smape.html) | \(\mathrm{sMAPE} = \frac{100}{n}\sum_i\frac{2\lvert o_i-p_i\rvert}{\lvert o_i\rvert+\lvert p_i\rvert}\) | 0 | Symmetric absolute percentage error (0--200%). | [@hyndman2006] | | **MSLE** [`msle()`](../reference/msle.html) | \(\mathrm{MSLE} = \frac{1}{n}\sum_i[\log(1+o_i)-\log(1+p_i)]^2\) | 0 | Squared error on the log1p scale. Requires non-negative values. | [@hodson2022] | | **RMSLE** [`rmsle()`](../reference/rmsle.html) | \(\mathrm{RMSLE} = \sqrt{\mathrm{MSLE}}\) | 0 | Root mean squared error on the log1p scale. | [@hodson2022] | | **RAE** [`rae()`](../reference/rae.html) | \(\mathrm{RAE} = \frac{\sum_i\lvert o_i-p_i\rvert}{\sum_i\lvert o_i-\bar{o}\rvert}\) | 0 | Absolute error relative to predicting the observed mean; 1 is the benchmark. | [@hyndman2006] | | **RRMSE** [`rrmse()`](../reference/rrmse.html) | \(\mathrm{RRMSE} = 100\,\mathrm{RMSE}/\lvert\bar{o}\rvert\) | 0 | RMSE as a percentage of the absolute observed mean. | [@willmott1985] | | **Willmott's \(d\)** [`willmott_d()`](../reference/willmott_d.html) | \(d = 1-\frac{\sum_i(o_i-p_i)^2}{\sum_i(\lvert p_i-\bar{o}\rvert+\lvert o_i-\bar{o}\rvert)^2}\) | 1 | Index of agreement; sensitive to large errors. | [@willmott1985] | | **KGE (2009)** [`kge()`](../reference/kge.html) | \(\mathrm{KGE} = 1-\sqrt{(r-1)^2+(s_p/s_o-1)^2+(\bar{p}/\bar{o}-1)^2}\) | 1 | Combines correlation, variability ratio, and mean ratio. | [@gupta2009] | `MAPE`, `MPE`, `sMAPE`, and `RRMSE` are returned as percentages. `sMAPE` ranges from 0 to 200%; the other percentage metrics are unbounded above. Percentage-based metrics require particular care when observations or their mean are zero or close to zero. # 10. Quantile predictions [`pinball_loss()`](../reference/pinball_loss.html) is not included in `model_metrics(extended = TRUE)` because it requires a specific quantile level. For a quantile prediction \(q_\tau\), the loss depends on the requested probability \(\tau\). For example, at the median: ```{r pinball-median} pinball_loss( obs, models$Good, level = 0.5 ) ``` At `level = 0.5`, pinball loss is one-half of MAE. For a 90th-percentile prediction: ```{r pinball-90, eval=FALSE} pinball_loss( obs, predicted_q90, level = 0.90 ) ``` The quantile level should be chosen according to the prediction being evaluated. # 11. How should metrics be combined? A useful general-purpose evaluation of continuous predictions should usually contain metrics from different categories. For example: ```{r complementary-set} data.frame( bias = bias(obs, models$Good), mae = mae(obs, models$Good), rmse = rmse(obs, models$Good), correlation = correlation(obs, models$Good), R2 = R2(obs, models$Good), ccc = ccc(obs, models$Good) ) ``` This combination answers several distinct questions: * `bias()` — is there systematic overprediction or underprediction? * `mae()` — what is the typical absolute error? * `rmse()` — what is the squared-error-weighted error magnitude? * `correlation()` — is the observed pattern reproduced? * `R2()` — does the model outperform the observed-mean benchmark? * `ccc()` — how closely do predicted values agree with observations overall? The precise set should depend on the application. There is no universal threshold at which RMSE, correlation, CCC, KGE (2009), or another metric automatically becomes "good". # 12. Related and redundant metrics Several available statistics are mathematically related. For example: \[ \mathrm{MSE}=\mathrm{RMSE}^2, \] and, when defined, \[ \mathrm{RPD} = \frac{1}{\mathrm{NRMSE}}. \] SEP and cRMSE also contain essentially the same centred error information but use different divisors. Similarly, `R2()`, `nse()`, and `mec()` are three names for exactly the same statistic in `modelskill`. These functions remain available because different scientific communities use different conventions, but reporting several mathematically equivalent metrics does not provide independent evidence about model performance. # 13. Missing values and special cases By default, `modelskill` removes incomplete observation-prediction pairs separately for each model. For example: ```{r missing-values} obs_missing <- obs pred_missing <- models$Good pred_missing[c(5, 20)] <- NA rmse( obs_missing, pred_missing ) ``` For fair comparison among several models, it is generally preferable that all models are evaluated on the same validation observations. Some metrics are undefined in particular situations: * Pearson correlation and `r2()` are undefined for constant vectors; * `R2()` / NSE / MEC are undefined when the observations are constant; * percentage metrics can be undefined or unstable when their denominators are zero or close to zero; * MSLE and RMSLE require non-negative observations and predictions; * KGE (2009) requires variation in both observations and predictions and a non-zero observed mean. The individual function documentation describes these cases in detail. # 14. Practical recommendations For most prediction-validation analyses: 1. **Inspect observations versus predictions first.** Summary metrics can hide structure visible in the raw comparison. 2. **Report error magnitude.** MAE and/or RMSE provide interpretable error measures in response units. 3. **Report systematic error.** Bias indicates whether predictions are systematically too high or too low. 4. **Separate association from accuracy.** Correlation and lowercase `r2()` measure association and can remain high despite substantial prediction bias. 5. **Use an agreement or efficiency measure when useful.** CCC measures agreement, while uppercase `R2()` / NSE / MEC compares model performance with the observed-mean benchmark. 6. **Use specialised metrics only when their interpretation fits the problem.** Percentage errors, RPD, RPIQ, KGE (2009), and quantile loss can be useful, but should not automatically be reported for every application. 7. **Do not select a model from one metric alone.** Different metrics describe different aspects of prediction quality. # 15. References * Bellon-Maurel, V., Fernandez-Ahumada, E., Palagos, B., Roger, J.-M., and McBratney, A. (2010). Critical review of chemometric indicators commonly used for assessing the quality of the prediction of soil attributes by NIR spectroscopy. *Trends in Analytical Chemistry*, 29, 1073-1081. * Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F. (2009). Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling. *Journal of Hydrology*, 377, 80-91. * Hodson, T. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not. *Geoscientific Model Development*, 15, 5481-5487. * Hyndman, R. J. and Koehler, A. B. (2006). Another look at measures of forecast accuracy. *International Journal of Forecasting*, 22, 679-688. * Janssen, P. H. M. and Heuberger, P. S. C. (1995). Calibration of process-oriented models. *Ecological Modelling*, 83, 55-66. * Koenker, R. and Bassett, G. (1978). Regression quantiles. *Econometrica*, 46, 33-50. * Legates, D. R. and McCabe, G. J. (1999). Evaluating the use of goodness-of-fit measures in hydrologic and hydroclimatic model validation. *Water Resources Research*, 35, 233-241. * Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. *Biometrics*, 45, 255-268. * Nash, J. E. and Sutcliffe, J. V. (1970). River flow forecasting through conceptual models part I: A discussion of principles. *Journal of Hydrology*, 10, 282-290. * Taylor, K. E. (2001). Summarizing multiple aspects of model performance in a single diagram. *Journal of Geophysical Research*, 106, 7183-7192. * Willmott, C. J., Ackleson, S. G., Davis, R. E., Feddema, J. J., Klink, K. M., Legates, D. R., O'Donnell, J., and Rowe, C. M. (1985). Statistics for the evaluation and comparison of models. *Journal of Geophysical Research*, 90, 8995-9005. * Willmott, C. J. and Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. *Climate Research*, 30, 79-82. # 16. Next steps Use the [predictive-uncertainty evaluation](predictive-uncertainty.html) article when models provide prediction intervals, quantiles, standard deviations, or complete predictive distributions. Use the [summary diagrams and diagnostic plots](summary-diagrams.html) article to compare several models graphically using solar, target, and Taylor diagrams.