Package {accuracylevel}

accuracylevel: Robust Accuracy-Level Metrics for Predictive Model Evaluation

Description

The accuracylevel package implements novel accuracy-level metrics for evaluating continuous data prediction models. These metrics offer robust, consistent, and interpretable evaluation on a 0–100\ limitations of conventional metrics like RMSE, MAE, and MAPE.

Main Functions

accuracy_level

Compute all accuracy-level metrics (CSE, CAE, CAPE, SCAPE).

cse, cae, cape, scape

Individual metric convenience functions.

Threshold Functions

calculate_threshold

Calculate error thresholds from a baseline model.

auto_threshold

Automatic quartile selection based on a target APE.

Integration

caret_summary

caret package integration.

cse_l1, cae_l1, etc.

tidymodels / yardstick integration.

al_forecast_accuracy

forecast package integration.

Comparison

conventional_metrics

RMSE, MAE, MAPE, etc.

robust_metrics

MedAE, Huber, trimmed MSE, etc.

compare_all_metrics

Side-by-side comparison.

Author(s)

Maintainer: Achmad Syahrul Choir madsyair@stis.ac.id

Authors:

• Achmad Syahrul Choir madsyair@stis.ac.id

• Mety Agustini mety.assahid@bps.go.id

• Kartika Fithriasari kartika_f@statistika.its.ac.id

• Dedy Dwi Prastyo dedy-dp@statistika.its.ac.id

References

Agustini, M., Fithriasari, K., & Prastyo, D.D. (2026). An accuracy-level method for robust evaluation in predictive analytics. Decision Analytics Journal, 18, 100661. doi:10.1016/j.dajour.2025.100661

See Also

Useful links:

• https://github.com/madsyair/accuracylevel

• Report bugs at https://github.com/madsyair/accuracylevel/issues

Calculate Absolute Error

Description

Compute absolute error between actual and predicted values, as defined in Equation (2) of Agustini et al. (2026).

Usage

absolute_error(actual, predicted, na.rm = FALSE)

Arguments

Value

Numeric vector of absolute errors.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
absolute_error(actual, predicted)

Calculate Absolute Percentage Error

Description

Compute absolute percentage error between actual and predicted values, as defined in Equation (3) of Agustini et al. (2026). Note that the result is on the proportion scale (not multiplied by 100).

Usage

absolute_percentage_error(actual, predicted, na.rm = FALSE)

Arguments

Value

Numeric vector of absolute percentage errors. Returns Inf for observations where actual is zero.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
absolute_percentage_error(actual, predicted)

Compute Accuracy-Level Metrics

Description

Calculate accuracy-level metrics (CSE, CAE, CAPE, SCAPE) for evaluating prediction model performance. These metrics assess the proportion of observations falling within predefined error threshold levels, providing a robust and interpretable evaluation on a 0–100\ scale.

Usage

accuracy_level(
  actual,
  predicted,
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL,
  na.rm = FALSE
)

Arguments

Details

The accuracy-level method introduces four metrics:

CSE (Counted Squared Error)

Proportion of observations within squared error threshold levels.

CAE (Counted Absolute Error)

Proportion of observations within absolute error threshold levels.

CAPE (Counted Absolute Percentage Error)

Proportion of observations within absolute percentage error threshold levels.

SCAPE (Symmetric Counted Absolute Percentage Error)

Proportion of observations within symmetric absolute percentage error threshold levels.

For each metric, four accuracy levels are defined:

• Level 1: \varepsilon < T (highest accuracy)

• Level 2: T \le \varepsilon < 2T

• Level 3: 2T \le \varepsilon < 5T

• Level 4: \varepsilon \ge 5T (lowest accuracy)

where T is the base threshold determined from the baseline model's error distribution. Crucially, thresholds for CSE, CAE, CAPE, and SCAPE are each derived from the same quartile of the baseline model's SE, AE, APE, and sAPE respectively (Figure 2 of the paper).

Edge cases are handled as follows: observations with a non-finite error (for example APE when actual is zero, or sAPE when both actual and predicted are zero) are assigned to Level 4. When the baseline threshold T equals zero (a perfectly fitting baseline), a machine-epsilon boundary is used so that exact-zero errors fall into Level 1.

Value

An object of class "accuracy_level" with elements: metrics (data frame with accuracy percentages for each level and metric type), mean_errors (data frame with mean errors per level), threshold (the primary threshold object used), thresholds_all (per-error-type threshold objects), n_obs (number of observations), and counts (count of observations at each level).

References

Agustini, M., Fithriasari, K., & Prastyo, D.D. (2026). An accuracy-level method for robust evaluation in predictive analytics. Decision Analytics Journal, 18, 100661. doi:10.1016/j.dajour.2025.100661

See Also

calculate_threshold, auto_threshold, cse, cae, cape, scape

Examples

# ---- Paper Table 4: Simple case ----
actual <- c(7, 6.03, 2.02, 5.1, 9, 1, 3, 4.38, 1, 8.07)
m1 <- c(6.05, 5.02, 1.32, 5.15, 8, 2.2, 2.7, 3.48, 1, 7.56)
m3 <- c(7.01, 6.04, 2.09, 5.11, 9.01, 5.1, 3.01, 4.39, 1, 8.1)

# Model 1 as baseline, Q2
thresh <- calculate_threshold(actual, m1, quartile = 2)

# Evaluate Model 3 (paper expects 90% at L1)
result <- accuracy_level(actual, m3, threshold = thresh)
print(result)

Full Accuracy-Level Metrics for yardstick

Description

Full Accuracy-Level Metrics for yardstick

Usage

accuracy_level_metrics(data, truth, estimate, na_rm = TRUE, ...)

## S3 method for class 'data.frame'
accuracy_level_metrics(data, truth, estimate, na_rm = TRUE, ...)

Arguments

Value

A tibble with 16 rows (4 metrics x 4 levels).

Examples

if (requireNamespace("rlang", quietly = TRUE)) {
  df <- data.frame(truth = c(10, 20, 30, 40, 50),
                   estimate = c(11, 19, 32, 38, 51))
  accuracy_level_metrics(df, truth, estimate)
}

Compare Multiple Forecast Models

Description

Compare Multiple Forecast Models

Usage

al_compare_forecasts(
  ...,
  test = NULL,
  metric = c("cae", "cape", "cse", "scape"),
  threshold = NULL
)

Arguments

Value

A list with optimal_model, comparison table, and full_results.

Examples

actual <- c(10, 20, 30, 40, 50)
res <- al_compare_forecasts(
  A = list(forecast = c(11, 19, 32, 38, 51), test = actual),
  B = list(forecast = c(15, 25, 35, 45, 55), test = actual)
)
res$comparison

Extended Forecast Accuracy Summary

Description

Extended Forecast Accuracy Summary

Usage

al_extended_accuracy(forecast_obj, test, threshold = NULL)

Arguments

Value

A data frame combining traditional and accuracy-level metrics.

Examples

pred <- c(11, 19, 32, 38, 51)
actual <- c(10, 20, 30, 40, 50)
al_extended_accuracy(pred, actual)

Accuracy-Level Metrics for Forecast Objects

Description

Calculate accuracy-level metrics for forecast objects from the forecast package, or for plain numeric predictions.

Usage

al_forecast_accuracy(object, test, threshold = NULL)

## Default S3 method:
al_forecast_accuracy(object, test, threshold = NULL)

## S3 method for class 'forecast'
al_forecast_accuracy(object, test, threshold = NULL)

Arguments

Value

An accuracy_level object.

Examples

# With plain numeric vectors
pred <- c(11, 19, 32, 38, 51)
actual <- c(10, 20, 30, 40, 50)
al_forecast_accuracy(pred, actual)

Create Metric Set for tidymodels

Description

Create Metric Set for tidymodels

Usage

al_metric_set(include_traditional = TRUE)

Arguments

Value

A metric set function.

Note

The level-1 metrics in the set derive their error thresholds from the data being evaluated (a self-referential baseline). In resampling or cross-validation, each fold therefore uses its own threshold, which limits strict cross-fold comparability. For a fixed baseline across folds, evaluate with accuracy_level using a pre-computed al_threshold object (see calculate_threshold). Case weights are not supported; any case_weights passed by tune are ignored.

Examples

if (requireNamespace("yardstick", quietly = TRUE)) {
  al_metrics <- al_metric_set()
}

Time Series Cross-Validation with Accuracy-Level Metrics

Description

Time Series Cross-Validation with Accuracy-Level Metrics

Usage

al_tsCV(
  y,
  forecastfunction,
  h = 1,
  initial = 10,
  window = NULL,
  metric = c("cae", "cape", "cse", "scape"),
  ...
)

Arguments

Value

A data frame of class "al_tsCV" with per-fold results.

Examples

ma_fc <- function(x, h) rep(mean(x), h)
y <- sin(seq(0, 4 * pi, length.out = 50)) * 10 + 50
res <- al_tsCV(y, ma_fc, h = 5, initial = 20)
print(res)

Automatic Threshold Selection

Description

Automatically select the best quartile for threshold calculation based on the absolute percentage error approaching a target value (default 0.1), following the recommendation in Section 3.4.5 of Agustini et al. (2026).

Usage

auto_threshold(
  actual,
  predicted,
  target_ape = 0.1,
  error_type = "ape",
  multipliers = c(2, 5)
)

Arguments

Value

An al_threshold object with automatically selected quartile.

Examples

actual <- c(10, 20, 30, 40, 50, 60, 70, 80, 90, 100)
predicted <- c(11, 19, 32, 38, 51, 58, 72, 78, 92, 98)
thresh <- auto_threshold(actual, predicted)
print(thresh)

Counted Absolute Error (CAE)

Description

Counted Absolute Error (CAE)

Usage

cae(
  actual,
  predicted,
  level = 1,
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL,
  as_decimal = FALSE
)

Arguments

Value

Numeric scalar.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
cae(actual, predicted, level = 1)

CAE Level 1 Metric for yardstick

Description

CAE Level 1 Metric for yardstick

Usage

cae_l1(data, truth, estimate, na_rm = TRUE, ...)

## S3 method for class 'data.frame'
cae_l1(data, truth, estimate, na_rm = TRUE, ...)

Arguments

Value

A tibble.

Calculate Error Thresholds from a Baseline Model

Description

Calculate error threshold levels based on a baseline model's error distribution. The threshold is determined using quartiles of the specified error type, following the procedure in Figure 2 of Agustini et al. (2026).

Quartiles are computed using the inverse empirical CDF (R's type = 1), consistent with the paper.

Usage

calculate_threshold(
  actual,
  predicted,
  error_type = c("ape", "sape", "se", "ae"),
  quartile = 2,
  multipliers = c(2, 5)
)

Arguments

Value

A list of class "al_threshold" with elements: threshold (the base threshold value T), levels (a list with L1–L4 boundary pairs), error_type (the error type used), quartile (the quartile used), multipliers (the multiplier values), and baseline_quartiles (a named list with the selected quartile value for every error type: se, ae, ape, sape – this is the key element that lets accuracy_level derive thresholds for all four metrics from a single baseline model).

References

Agustini, M., Fithriasari, K., & Prastyo, D.D. (2026). An accuracy-level method for robust evaluation in predictive analytics. Decision Analytics Journal, 18, 100661. doi:10.1016/j.dajour.2025.100661

See Also

accuracy_level, auto_threshold

Examples

# --- Paper Table 4: simple case, Model 1 as baseline ---
actual  <- c(7, 6.03, 2.02, 5.1, 9, 1, 3, 4.38, 1, 8.07)
model1  <- c(6.05, 5.02, 1.32, 5.15, 8, 2.2, 2.7, 3.48, 1, 7.56)

# Q2 of APE ~ 0.1111, close to the target 0.10
thresh <- calculate_threshold(actual, model1, quartile = 2)
print(thresh)

# Stricter thresholds via Q1
thresh_q1 <- calculate_threshold(actual, model1, quartile = 1)

Counted Absolute Percentage Error (CAPE)

Description

Counted Absolute Percentage Error (CAPE)

Usage

cape(
  actual,
  predicted,
  level = 1,
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL,
  as_decimal = FALSE
)

Arguments

Value

Numeric scalar.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
cape(actual, predicted, level = 1)

CAPE Level 1 Metric for yardstick

Description

CAPE Level 1 Metric for yardstick

Usage

cape_l1(data, truth, estimate, na_rm = TRUE, ...)

## S3 method for class 'data.frame'
cape_l1(data, truth, estimate, na_rm = TRUE, ...)

Arguments

Value

A tibble.

Create Single Metric caret Summary

Description

Create Single Metric caret Summary

Usage

caret_single_metric(
  metric_type = c("cse", "cae", "cape", "scape"),
  level = 1,
  threshold = NULL
)

Arguments

Value

A function suitable for summaryFunction in caret::trainControl.

Examples

if (requireNamespace("caret", quietly = TRUE)) {
  fn <- caret_single_metric("cae", level = 1)
}

Create Custom caret Metrics

Description

Create a summary function for use with caret's trainControl. Returns L1 accuracy for all four metrics plus traditional metrics.

Usage

caret_summary(threshold = NULL)

Arguments

Value

A function suitable for summaryFunction in caret::trainControl.

Examples

if (requireNamespace("caret", quietly = TRUE)) {
  al_summary <- caret_summary()
}

Create Extended caret Summary with All Levels

Description

Create Extended caret Summary with All Levels

Usage

caret_summary_extended(threshold = NULL)

Arguments

Value

A function suitable for summaryFunction in caret::trainControl.

Examples

if (requireNamespace("caret", quietly = TRUE)) {
  al_ext <- caret_summary_extended()
}

Compare All Metric Types

Description

Comprehensive comparison of conventional, robust, and accuracy-level metrics.

Usage

compare_all_metrics(actual, predicted, threshold = NULL)

Arguments

Value

A list of class "metrics_comparison" with conventional, robust, accuracy_level, and summary elements.

Examples

actual <- c(10, 20, 30, 40, 50, 60, 70, 80, 90, 100)
predicted <- c(11, 19, 32, 38, 51, 58, 72, 78, 92, 98)
result <- compare_all_metrics(actual, predicted)
print(result)

Compare Multiple Models

Description

Compare multiple prediction models using accuracy-level metrics and identify the optimal one following the model-selection procedure in Figure 3 of Agustini et al. (2026).

Usage

compare_models(
  ...,
  metric = c("cse", "cae", "cape", "scape"),
  threshold = NULL
)

Arguments

Details

The optimal model is selected by the Figure 3 algorithm:

1. Compare the Level 1 accuracy across models; the model with the highest value is selected.

2. If two or more models tie on Level 1 accuracy, the tie is broken using the mean error (ME) of the corresponding level (lower ME is better).

3. If the ME values are also equal, the comparison proceeds to the next accuracy level, repeating until the optimal model is identified.

Earlier releases used the simpler rule of ranking by Level 1 then Level 2 accuracy; this version implements the full ME-based tie-break.

Value

A list with optimal_model, comparison table (accuracy and mean error per level), metric_used, and full_results.

Examples

actual <- c(7, 6.03, 2.02, 5.1, 9, 1, 3, 4.38, 1, 8.07)
m1 <- list(actual = actual,
           predicted = c(6.05, 5.02, 1.32, 5.15, 8, 2.2, 2.7, 3.48, 1, 7.56))
m3 <- list(actual = actual,
           predicted = c(7.01, 6.04, 2.09, 5.11, 9.01, 5.1, 3.01, 4.39, 1, 8.1))

res <- compare_models(Model1 = m1, Model3 = m3, metric = "cape")
print(res$comparison)
res$optimal_model

Calculate Conventional Metrics

Description

Compute commonly used evaluation metrics including R-squared, RMSE, NRMSE, MAE, MAPE, and SMAPE.

Usage

conventional_metrics(actual, predicted, na.rm = FALSE)

Arguments

Details

NRMSE is normalised by the mean of actual (returned as NA when that mean is zero). R_squared is the usual 1 - SS_{res}/SS_{tot} and may be negative for models worse than the mean (unlike the 0–1 range quoted in Table 1 of the paper). MAPE and SMAPE are returned on the percentage scale and ignore non-finite per-observation terms (e.g. division by a zero actual value).

Value

A named numeric vector with six elements.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
conventional_metrics(actual, predicted)

Counted Squared Error (CSE)

Description

Compute the Counted Squared Error metric at a specified level.

Usage

cse(
  actual,
  predicted,
  level = 1,
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL,
  as_decimal = FALSE
)

Arguments

Value

Numeric scalar: the percentage (or proportion) of observations at the specified accuracy level.

Examples

actual <- c(7, 6.03, 2.02, 5.1, 9, 1, 3, 4.38, 1, 8.07)
predicted <- c(6.05, 5.02, 1.32, 5.15, 8, 2.2, 2.7, 3.48, 1, 7.56)

cse(actual, predicted, level = 1)
sapply(1:4, function(l) cse(actual, predicted, level = l))

CSE Level 1 Metric for yardstick

Description

CSE Level 1 Metric for yardstick

Usage

cse_l1(data, truth, estimate, na_rm = TRUE, ...)

## S3 method for class 'data.frame'
cse_l1(data, truth, estimate, na_rm = TRUE, ...)

Arguments

Value

A tibble with .metric, .estimator, .estimate.

Examples

if (requireNamespace("rlang", quietly = TRUE)) {
  df <- data.frame(truth = c(10, 20, 30), estimate = c(11, 19, 28))
  cse_l1(df, truth, estimate)
}

Get All Levels for a Metric

Description

Convenience function to obtain all four levels at once.

Usage

get_all_levels(
  actual,
  predicted,
  metric = c("cse", "cae", "cape", "scape"),
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL
)

Arguments

Value

Named numeric vector of length 4.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
get_all_levels(actual, predicted, "cae")

Print Method for al_threshold Objects

Description

Print Method for al_threshold Objects

Usage

## S3 method for class 'al_threshold'
print(x, ...)

Arguments

Value

Invisibly returns the input object.

Calculate Robust Metrics

Description

Compute robust evaluation metrics including Median Absolute Error, Trimmed Mean Squared Error, Huber Loss, and Quantile Loss.

Usage

robust_metrics(
  actual,
  predicted,
  trim_percent = 0.1,
  huber_delta = 1,
  tau = 0.5,
  na.rm = FALSE
)

Arguments

Value

A named numeric vector with four elements.

Examples

actual <- c(10, 20, 30, 40, 50, 1000)
predicted <- c(11, 19, 32, 38, 51, 60)
robust_metrics(actual, predicted)

Symmetric Counted Absolute Percentage Error (SCAPE)

Description

Symmetric Counted Absolute Percentage Error (SCAPE)

Usage

scape(
  actual,
  predicted,
  level = 1,
  threshold = NULL,
  baseline_actual = NULL,
  baseline_predicted = NULL,
  as_decimal = FALSE
)

Arguments

Value

Numeric scalar.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
scape(actual, predicted, level = 1)

SCAPE Level 1 Metric for yardstick

Description

SCAPE Level 1 Metric for yardstick

Usage

scape_l1(data, truth, estimate, na_rm = TRUE, ...)

## S3 method for class 'data.frame'
scape_l1(data, truth, estimate, na_rm = TRUE, ...)

Arguments

Value

A tibble.

Calculate Squared Error

Description

Compute squared error between actual and predicted values, as defined in Equation (1) of Agustini et al. (2026).

Usage

squared_error(actual, predicted, na.rm = FALSE)

Arguments

Value

Numeric vector of squared errors.

References

Agustini, M., Fithriasari, K., & Prastyo, D.D. (2026). An accuracy-level method for robust evaluation in predictive analytics. Decision Analytics Journal, 18, 100661. doi:10.1016/j.dajour.2025.100661

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
squared_error(actual, predicted)

Calculate Symmetric Absolute Percentage Error

Description

Compute symmetric absolute percentage error between actual and predicted values, as defined in Equation (4) of Agustini et al. (2026). Note that the result is on the proportion scale (not multiplied by 100).

Usage

symmetric_absolute_percentage_error(actual, predicted, na.rm = FALSE)

Arguments

Value

Numeric vector of symmetric absolute percentage errors. Returns NaN when both actual and predicted are zero.

Examples

actual <- c(10, 20, 30, 40, 50)
predicted <- c(11, 19, 32, 38, 51)
symmetric_absolute_percentage_error(actual, predicted)