1. Why use several prediction metrics?

Continuous predictions may be generated using machine-learning algorithms, statistical models, geostatistical methods, or process-based models. modelskill provides tools to evaluate the agreement between paired observations and predictions using a range of complementary performance metrics.

Predictive performance is inherently multidimensional. A model may exhibit low overall error while retaining systematic bias, reproduce the observed pattern while consistently over- or underpredicting, or achieve good aggregate accuracy while failing to capture the observed variability.

Accordingly, model evaluation should generally rely on a combination of complementary metrics rather than a single summary statistic, so that different aspects of predictive performance can be assessed explicitly.

Performance metrics should be computed using independent validation data whenever possible, or using an appropriate resampling or cross-validation procedure when an independent validation dataset is not available.

2. Example data

We create three deliberately different prediction models.

library(modelskill)

set.seed(123)

n <- 100

obs <- seq(0, 10, length.out = n) +
  rnorm(n, sd = 1)

models <- list(
  Good = obs + rnorm(n, sd = 0.5),

  # Constant positive offset
  Biased = obs + 1,

  # Larger random errors
  Noisy = obs + rnorm(n, sd = 2)
)

The models have simple characteristics:

• Good has relatively small random errors;
• Biased systematically overpredicts by one unit;
• Noisy has larger random errors.

3. Start by looking at the predictions

Before calculating summary statistics, it is useful to inspect observed versus predicted values.

plot_data <- data.frame(
  observed = rep(obs, times = length(models)),
  predicted = unlist(models, use.names = FALSE),
  model = rep(names(models), each = length(obs))
)

ggplot2::ggplot(
  plot_data,
  ggplot2::aes(
    x = observed,
    y = predicted,
    colour = model
  )
) +
  ggplot2::geom_abline(
    slope = 1,
    intercept = 0,
    linetype = "dashed"
  ) +
  ggplot2::geom_point(
    alpha = 0.7,
    size = 1.5
  ) +
  ggplot2::coord_equal() +
  ggplot2::theme_classic() +
  ggplot2::labs(
    x = "Observed",
    y = "Predicted",
    colour = "Model"
  )

The dashed 1:1 line represents perfect agreement. Good remains relatively close to this line, Biased is systematically shifted upward, and Noisy is more dispersed.

4. Calculate individual metrics

Every prediction metric in modelskill can be calculated separately. This is useful when only a few specific aspects of performance are required.

All individual functions use the same basic argument order:

metric(obs, pred)

where obs is the vector of observations and pred is the corresponding prediction vector.

rmse(
  obs,
  models$Good
)
#> [1] 0.4840658

rmse(
  obs,
  models$Noisy
)
#> [1] 1.905528

rmse(obs, models$Noisy)
#> [1] 1.905528
mae(obs, models$Noisy)
#> [1] 1.513235

bias(
  obs,
  models$Biased
)
#> [1] -1

5. A particularly important distinction: r2() versus R2()

modelskill contains two statistics whose names differ only by capitalization, but they measure different things.

r2(
  obs,
  models$Biased
)
#> [1] 1

R2(
  obs,
  models$Biased
)
#> [1] 0.8974284

c(
  R2 = R2(obs, models$Good),
  NSE = nse(obs, models$Good),
  MEC = mec(obs, models$Good)
)
#>        R2       NSE       MEC 
#> 0.9759654 0.9759654 0.9759654

correlation(
  obs,
  models$Biased
)
#> [1] 1

ccc(
  obs,
  models$Biased
)
#> [1] 0.9512161

6. Compare several models at once

When several models are being evaluated, model_metrics() calculates the main prediction metrics together.

model_metrics(
  models,
  obs,
  digits = 3
)
#>         model   bias   mae   mse  rmse nrmse crmse correlation    r2    R2
#> Good     Good  0.054 0.382 0.234 0.484 0.154 0.481       0.989 0.977 0.976
#> Biased Biased -1.000 1.000 1.000 1.000 0.319 0.000       1.000 1.000 0.897
#> Noisy   Noisy -0.241 1.513 3.631 1.906 0.607 1.890       0.866 0.749 0.628
#>        sd_ratio   ccc    Cb
#> Good      1.023 0.988 1.000
#> Biased    1.000 0.951 0.951
#> Noisy     1.206 0.849 0.980

The default output contains complementary information about:

• systematic error;
• average error magnitude;
• centred error;
• association;
• model efficiency;
• variability;
• concordance.

For the example data, no single column should be used to identify the “best” model without considering what aspect of prediction quality matters for the application.

7. Categories of prediction performance metrics

The following classification is intended as a practical guide. Some metrics are mathematically related and should not be interpreted as independent evidence.

There is no universally best metric. The appropriate combination depends on the scientific objective and the consequences of different types of prediction error.

8. Core metrics

The default output of model_metrics() contains the principal metrics used for continuous prediction evaluation.

For the retained observation-prediction pairs, let \(o_i\) denote observations, \(p_i\) predictions, and \(n\) the number of pairs. Let \(\bar{o}\) and \(\bar{p}\) denote their means.

Here \(s_o\) and \(s_p\) are the sample standard deviations of observations and predictions. For CCC, \(v_o\), \(v_p\), and \(c_{op}\) are the corresponding population variances and covariance used by the package.

The ranges and special cases of each metric are described in its linked reference page.

9. Extended metrics

Additional metrics are available when a more specialised evaluation is required.

model_metrics(
  models,
  obs,
  extended = TRUE,
  digits = 3
)
#>         model   bias   mae   mse  rmse nrmse crmse correlation    r2    R2
#> Good     Good  0.054 0.382 0.234 0.484 0.154 0.481       0.989 0.977 0.976
#> Biased Biased -1.000 1.000 1.000 1.000 0.319 0.000       1.000 1.000 0.897
#> Noisy   Noisy -0.241 1.513 3.631 1.906 0.607 1.890       0.866 0.749 0.628
#>        sd_ratio   ccc    Cb  mdae   rpd   rpiq   sep    rer    mape     mpe
#> Good      1.023 0.988 1.000 0.312 6.483 11.063 0.483 25.709  17.597   1.271
#> Biased    1.000 0.951 0.951 1.000 3.138  5.355 0.000 12.445  55.918 -25.264
#> Noisy     1.206 0.849 0.980 1.213 1.647  2.810 1.900  6.531 105.193  -7.746
#>         smape msle rmsle   rae  rrmse willmott_d   kge
#> Good   18.659   NA    NA 0.144  9.509      0.994 0.972
#> Biased 33.766   NA    NA 0.377 19.645      0.975 0.804
#> Noisy  49.121   NA    NA 0.570 37.434      0.920 0.750

The extended output adds robust error measures, scale-normalised metrics, percentage errors, agreement measures, and KGE (2009).

These metrics should be selected because they answer a relevant scientific question, not simply because they are available.

For example, the median absolute error is less affected by a few unusually large errors than RMSE:

mdae(
  obs,
  models$Noisy
)
#> [1] 1.212603

KGE (2009), the original Kling-Gupta efficiency formulation, combines correlation, variability, and mean agreement:

kge(
  obs,
  models$Good
)
#> [1] 0.9720975

10. Quantile predictions

pinball_loss() is not included in model_metrics(extended = TRUE) because it requires a specific quantile level.

For a quantile prediction \(q_\tau\), the loss depends on the requested probability \(\tau\).

For example, at the median:

pinball_loss(
  obs,
  models$Good,
  level = 0.5
)
#> [1] 0.1910524

At level = 0.5, pinball loss is one-half of MAE.

For a 90th-percentile prediction:

pinball_loss(
  obs,
  predicted_q90,
  level = 0.90
)

The quantile level should be chosen according to the prediction being evaluated.

11. How should metrics be combined?

A useful general-purpose evaluation of continuous predictions should usually contain metrics from different categories.

For example:

data.frame(
  bias = bias(obs, models$Good),
  mae = mae(obs, models$Good),
  rmse = rmse(obs, models$Good),
  correlation = correlation(obs, models$Good),
  R2 = R2(obs, models$Good),
  ccc = ccc(obs, models$Good)
)
#>        bias       mae      rmse correlation        R2     ccc
#> 1 0.0537734 0.3821047 0.4840658   0.9886633 0.9759654 0.98826

This combination answers several distinct questions:

• bias() — is there systematic overprediction or underprediction?
• mae() — what is the typical absolute error?
• rmse() — what is the squared-error-weighted error magnitude?
• correlation() — is the observed pattern reproduced?
• R2() — does the model outperform the observed-mean benchmark?
• ccc() — how closely do predicted values agree with observations overall?

The precise set should depend on the application. There is no universal threshold at which RMSE, correlation, CCC, KGE (2009), or another metric automatically becomes “good”.

12. Related and redundant metrics

Several available statistics are mathematically related.

For example:

\[ \mathrm{MSE}=\mathrm{RMSE}^2, \]

and, when defined,

\[ \mathrm{RPD} = \frac{1}{\mathrm{NRMSE}}. \]

SEP and cRMSE also contain essentially the same centred error information but use different divisors.

Similarly, R2(), nse(), and mec() are three names for exactly the same statistic in modelskill.

These functions remain available because different scientific communities use different conventions, but reporting several mathematically equivalent metrics does not provide independent evidence about model performance.

13. Missing values and special cases

By default, modelskill removes incomplete observation-prediction pairs separately for each model.

For example:

obs_missing <- obs
pred_missing <- models$Good

pred_missing[c(5, 20)] <- NA

rmse(
  obs_missing,
  pred_missing
)
#> [1] 0.4838553

For fair comparison among several models, it is generally preferable that all models are evaluated on the same validation observations.

Some metrics are undefined in particular situations:

• Pearson correlation and r2() are undefined for constant vectors;
• R2() / NSE / MEC are undefined when the observations are constant;
• percentage metrics can be undefined or unstable when their denominators are zero or close to zero;
• MSLE and RMSLE require non-negative observations and predictions;
• KGE (2009) requires variation in both observations and predictions and a non-zero observed mean.

The individual function documentation describes these cases in detail.

14. Practical recommendations

For most prediction-validation analyses:

1. Inspect observations versus predictions first.
   Summary metrics can hide structure visible in the raw comparison.

2. Report error magnitude.
   MAE and/or RMSE provide interpretable error measures in response units.

3. Report systematic error.
   Bias indicates whether predictions are systematically too high or too low.

4. Separate association from accuracy.
   Correlation and lowercase r2() measure association and can remain high despite substantial prediction bias.

5. Use an agreement or efficiency measure when useful.
   CCC measures agreement, while uppercase R2() / NSE / MEC compares model performance with the observed-mean benchmark.

6. Use specialised metrics only when their interpretation fits the problem.
   Percentage errors, RPD, RPIQ, KGE (2009), and quantile loss can be useful, but should not automatically be reported for every application.

7. Do not select a model from one metric alone.
   Different metrics describe different aspects of prediction quality.

15. References

• Bellon-Maurel, V., Fernandez-Ahumada, E., Palagos, B., Roger, J.-M., and McBratney, A. (2010). Critical review of chemometric indicators commonly used for assessing the quality of the prediction of soil attributes by NIR spectroscopy. Trends in Analytical Chemistry, 29, 1073-1081. https://doi.org/10.1016/j.trac.2010.05.006

• Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F. (2009). Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling. Journal of Hydrology, 377, 80-91. https://doi.org/10.1016/j.jhydrol.2009.08.003

• Hodson, T. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): when to use them or not. Geoscientific Model Development, 15, 5481-5487. https://doi.org/10.5194/gmd-15-5481-2022

• Hyndman, R. J. and Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22, 679-688. https://doi.org/10.1016/j.ijforecast.2006.03.001

• Janssen, P. H. M. and Heuberger, P. S. C. (1995). Calibration of process-oriented models. Ecological Modelling, 83, 55-66. https://doi.org/10.1016/0304-3800(95)00084-9

• Koenker, R. and Bassett, G. (1978). Regression quantiles. Econometrica, 46, 33-50. https://doi.org/10.2307/1913643

• Legates, D. R. and McCabe, G. J. (1999). Evaluating the use of goodness-of-fit measures in hydrologic and hydroclimatic model validation. Water Resources Research, 35, 233-241. https://doi.org/10.1029/1998WR900018

• Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255-268. https://doi.org/10.2307/2532051

• Nash, J. E. and Sutcliffe, J. V. (1970). River flow forecasting through conceptual models part I: A discussion of principles. Journal of Hydrology, 10, 282-290. https://doi.org/10.1016/0022-1694(70)90255-6

• Taylor, K. E. (2001). Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research, 106, 7183-7192. https://doi.org/10.1029/2000JD900719

• Willmott, C. J., Ackleson, S. G., Davis, R. E., Feddema, J. J., Klink, K. M., Legates, D. R., O’Donnell, J., and Rowe, C. M. (1985). Statistics for the evaluation and comparison of models. Journal of Geophysical Research, 90, 8995-9005. https://doi.org/10.1029/JC090iC05p08995

• Willmott, C. J. and Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30, 79-82. https://doi.org/10.3354/cr030079

16. Next steps

Use the predictive-uncertainty evaluation article when models provide prediction intervals, quantiles, standard deviations, or complete predictive distributions.

Use the summary diagrams and diagnostic plots article to compare several models graphically using solar, target, and Taylor diagrams.